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TITLE OF THE INVENTION 
MOVING IMAGE DECODING APPARATUS, MOVING IMAGE DECODING 
METHOD, IMAGE DECODING METHOD, AND IMAGE DECODING 

APPARATUS 

5 

FIELD OF THE INVENTION 

The present Invention relates to a technique for 
reclaiming a playback image by decoding all or some 
frcunes of moving image data, frames of which are 
10 independently encoded. 

BACKGROUND OF THE INVENTION 

In general, moving image data encoding methods 
are roughly classified based on whether or not to use 

15 inter-frame correlation. Those methods have merits and 
demerits , and which of methods is suitable depends on 
an application used. For example. Motion JPEG is a 
method of independently encoding each frame of moving 
image data as a single still image, and is an excunple 

20 of an encoding method that does not use any inter- frame 
correlation. As merits, this method allows easy moving 
image edit processes such as a division process, 
coupling process, partial rewrite process, and the like 
of moving images, and allows to select and decode 

25 freunes to be decoded in accordance with the processing 
performance on the decoding side, since frames are 
independently encoded. 
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In recent years, of the encoding methods that 
independently encode moving Image data for respective 
frames , a method of encoding each frame by combining 
wavelet transformation and bltplane encoding has 
5 received a lot of attention. Such moving Image 

encoding method has outstanding features: It allows 
decoding by changing the spatial resolution step by 
step by exploiting a subband decomposition scheme In 
wavelet transformation, and can change the decoding 

10 pixel precision step by step by changing the number of 
bltplanes to be decoded. 

JPEG2000 (ISO/IEC 15444) as an image encoding 
method that has been standardized by ISO/IEC 
JTC/SC29/WG1 is specified by a combination of wavelet 

15 transformation and bltplane encoding. Part 3 of that 
standard specifies a file format as Motion JPEG2000 
upon applying JPEG2000 to encoding of respective frames 
of a moving image. 

Such moving Image encoding methods represented by 

20 Motion JPEG2000 have merits such as flexibility in 

decoding resolution and decoding pixel precision, as 
described above, but have demerits such as a heavy load 
on an encoding/decoding process by bltplane encoding. 
Especially, when a video recorded by a dedicated moving 

25 image recording device is to be played back by a 

personal computer, all data cannot often be decoded and 
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displayed In real time depending on the performance of 
the computer. 

To solve such problem, methods that determine a 
decoding process time required to decode each frame, 
5 assign the decoding process time for respective 

encoding process units, and decode respective bltplanes 
within the assigned decoding process time have been 
disclosed (e.g., Japanese Patent Lald-Open 
Nos. 11-288307 and 2001-112004). 

10 However, with a moving Image decoding apparatus 

which aborts a decoding process after an elapse of a 
predetermined time, as disclosed In Japanese Patent 
Laid-open Nos. 11-288307, 2001-112004, and the like, 
since the decoding process abort points (the number of 

15 decoded bltplanes) readily vary for respective frames, 
a change In distortion pattern over time occurs upon 
playing back a moving Image, and appears as flickering, 
thus causing visual disturbance. 

20 SUMMARY OF THE INVENTION 

The present Invention has been made In 
consideration of the aforementioned problems, and has 
as Its object to provide a technique that can 
efficiently decode all or some frames of encoded moving 

25 Image data In correspondence with the processing 

performance of a moving Image decoding apparatus, and 



CFM03540US / P204-0121US 



can obtain high playback image quality with less visual 
disturbance . 

In order to achieve the above object, for example, 
a moving image decoding method of the present invention 
5 comprises the following arrangement. 

That is , a moving image decoding method of 
decoding encoded moving image data, which is generated 
by decomposing each frame of moving image data into a 
plurality of subbands, and encoding coefficients of the 
10 subbands from upper to lower bits for respective 

bitplanes or sub-bitplanes for each predetermined unit, 
comprising: 

a decoding process time information acquisition 
step of acquiring information used to examine a 

15 difference between a time assigned to a decoding 
process of encoded moving image data for the 
predetermined unit , and a time required for an actual 
decoding process; 

a non-decoding bitplane determination step of ^ 

20 determining bitplanes or sub-bitplanes which are not to 
be decoded on the basis of the information acquired in 
the decoding process time information acquisition step; 

a bitplane decoding step of reclaiming the 
coefficients of the subbands from encoded data of 

25 bitplanes or sub-bitplanes other than the bitplanes or 
sub-bitplanes determined in the non-decoding bitplane 
determination step; and 
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a subband composition step of generating frame 
data by compositing the coefficients of the plurality 
of subbands obtained in the bitplane decoding step. 

In order to achieve the above object, for example, 
5 a moving image decoding apparatus of the present 
invention comprises the following arrangement . 

That is , a moving image decoding apparatus for 
decoding encoded moving image data, which is generated 
by decomposing each frcune of moving image data into a 
10 plurality of subbands, and encoding coefficients of the 
subbands from upper to lower bits for respective 
bitplanes or sub-bitplanes for each predetermined unit, 
comprising: 

decoding process time information acquisition 
15 means for acquiring information used to examine a 
difference between a time assigned to a decoding 
process of encoded moving image data for the 
predetermined unit , and a time required for an actual 
decoding process; 
20 non-decoding bitplane determination means for 

determining bitplanes or sub-bitplanes which are not to 
be decoded on the basis of the information acquired by 
said decoding process time information acquisition 
means ; 

25 bitplane decoding means for reclaiming the 

coefficients of the subbands from encoded data of 
bitplanes or sub-bitplanes other than the bitplanes or 
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sub-bltplanes determined by said non-decoding bltplane 
determination means ; and 

subband composition means for generating frame 
data by compositing the coefficients of the plurality 
5 of subbands obtained by said bltplane decoding means • 

In order to achieve the above object, for example, 
an Image decoding method of the present Invention 
comprises the following arrangement. 

That Is, an Image decoding method of receiving 
10 moving Image data. In which images of respective frames 
have been encoded, and decoding encoded image data of 
the respective frames, comprising: 

a scunple frame decoding step of extracting 
encoded image data of a sample frame from the 
15 respective frames, and decoding the encoded image data 
using a predetermined decoding unit; 

a first measurement step of measuring a time 
required to decode the encoded image data of the sample 
frame ; 

20 a determination step of determining the number of 

decoding units to be decoded so that the time measured 
in the first measurement step becomes not more than a 
predetermined time; 
I a decoding step of decoding encoded image data of 

25 the respective frames in accordance with the determined 
number of decoding units; 
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a second measurement step of measuring a time 
required to decode each frame upon decoding the frame 
in the decoding step; 

an update step of accumulating a difference 
5 between the predetermined time and the time measured in 
the second measurement step every time the frame is 
decoded, and updating the number of decoding units 
determined in the determination step when the 
accumulated value becomes not less than a predetermined 
10 value; and 

in that the decoding step includes a step of 
decoding the encoded image data of each frame in 
accordance with the number of decoding units determined 
in the determination step or the number of decoding 
15 units updated in the update step. 

In order to achieve the above object, for example, 
an image decoding apparatus of the present invention 
comprises the following arrangement . 

That is, an image decoding apparatus for 
20 receiving moving image data, in which images of 

respective frames have been encoded, and decoding 
encoded image data of the respective frames , 
comprising: 

sample freune decoding means for extracting 
25 encoded image data of a S£unple frame from the 

respective frames , and decoding the encoded image data 
using a predetermined decoding unit; 
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first measurement means for measuring a time 
required to decode the encoded Image data of the sample 
frame; 

determination means for determining the number of 
5 decoding units to be decoded so that the time measured 
by the first measurement means becomes not more than a 
predetermined time; 

decoding means for decoding encoded Image data of 
the respective freunes In accordance with the determined 
10 number of decoding units; 

second measurement means for measuring a time 
required to decode each frame upon decoding the frame 
by the decoding means; 

update means for accumulating a difference 
15 between the predetermined time and the time measured by 
the second measurement means every time the frame Is 
decoded, and updating the number of decoding units 
determined by the determination means when the 
accumulated value becomes not less than a predetermined 
20 value; and 

In that the decoding means decodes the encoded 
Image data of each frame In accordance with the niomber 
of decoding units determined by the determination means 
or the number of decoding units updated by the update 
25 means . 

Other features and advantages of the present 
Invention will be apparent from the following 



- 8 - 



CFM03540US / P204-0121US 



description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
10 with the description, serve to explain the principles 
of the invention . 

Fig. 1 is a block diagram showing the arrangement 
of a moving image encoding apparatus in an embodiment 
of the present invention; 
15 Fig. 2 A is a view for explaining a subband of an 

image to be encoded, which is processed by 
two-dimensional (2D) discrete wavelet transformation; 

Fig. 2B is a view for explaining subbands of an 
image to be encoded, which is processed by 2D discrete 
20 wavelet transformation; 

Fig. 2C is a view for explaining subbands of an 
image to be encoded, which is processed by 2D discrete 
wavelet transf orroat ion ; 

Fig. 3 shows seven subbands obtained by two 2D 
25 discrete wavelet transformation processes; 
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F±g- 4 Is a flow chart for explaining the 
processing sequence for encoding subbands Sb by a 
bltplane encoder; 

Fig. 5 shows the detailed structure of a code 
5 sequence corresponding to encoded moving Image data for 
one frame, which Is generated by a code sequence 
f ormlng unit ; 

Fig. 6 shows an example of a code sequence of 
respective freunes stored In a secondary storage device; 
10 Fig. 7 Is a block diagram showing the arrangement 

of a moving Image decoding apparatus according to the 
first and third embodiments of the present Invention; 

Fig. 8 Is a table showing the relationship 
between Q factors and non-decoding bitplanes ND (Sb) of 
15 respective subbands or between Q factors and 

non- decoding pass counts NDP(Sb) of respective 
subbands ; 

Fig. 9 is a flow chart showing the flow of the 
process of the moving image decoding apparatus 
20 according to the first and third embodiments of the 
present invention; 

Fig. 10 is a block diagram showing the 
arrangement of a moving image decoding apparatus 
according to the second embodiment of the present 
25 invention; 
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Fig. 11 shows an example of a table held in a 
non- decoding bitplane determination unit according to 
the second embodiment of the present invention; 

Fig. 12 is a flow chart showing the flow of the 
5 process of the moving image decoding apparatus 

according to the second embodiment of the present 
invention; 

Fig. 13 is a flow chart showing the flow of the 
update process of an ND (Sb) table and SI in step 
10 S1107; 

Fig. 14 is a table showing the correspondence 
between subband indices SI and subbands according to 
the second embodiment of the present invention; 

Fig. 15 shows the structure of encoded moving 
15 image data for one frame, which is to be decoded by the 
moving image decoding apparatus according to the third 
embodiment of the present invention; 

Fig. 16 is a block diagram showing an example of 
the detailed arrangement of a moving image data output 
20 unit; 

Fig. 17 is a block diagram showing the 
arrangement of a moving image decoding apparatus 
according to the fourth embodiment of the present 
invention; 

25 Fig. 18 is a time chart showing in time series 

the operations of the moving image decoding apparatus 
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according to the fourth embodiment of the present 
invention; 

Fig. 19 is a flow chart showing the flow of the 
process of the moving image decoding apparatus 
5 according to the fourth embodiment of the present 
invention ; 

Fig. 20 is a table showing sample frame decoding 
times Dt'(Q) corresponding to Q factor values; 

Fig. 21 is a block diagram showing the functional 
10 arrangement of an image decoding apparatus according to 
the fifth embodiment of the present invention; 

Fig. 22 shows an example of the configuration of 
a table that shows the relationship between subband 
indices SI and decoding times (required decoding times) 
15 of subbands Sb; 

Fig. 23 shows an example of the configuration of 
a table that shows decoding flags F(Sb) corresponding 
to respective subbands Sb; 

Fig. 24 is a flow chart showing the flow of the 
20 decoding process of encoded moving image data by the 
image decoding apparatus according to the fifth 
embodiment of the present invention; 

Fig. 25 is a flow chart showing the flow of the 
decoding process of encoded moving image data by the 
25 image decoding apparatus according to the sixth 
embodiment of the present invention; and 
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Fig. 26 is a block diagram showing the basic 
arrangement of the image decoding apparatus according 
to the fifth embodiment of the present invention. 



5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
accompanying drawings. Note that the dimensions, 
materials, and shapes of building components, their 

10 relative layout, and the like described in the 

embodiments should be changed as needed depending on 
the arrangement of an apparatus to which the present 
invention is applied, and various conditions, and the 
present invention is not limited to their examples. 

15 <Outline of Encoded Moving Image Data> 

Encoded moving image data, which is to be decoded 
by a moving image decoding apparatus in an embodiment 
of the present invention, will be described first. 

Fig. 1 is a block diagram showing the arrangement 

20 of a moving image encoding apparatus 200 which 

generates encoded moving image data to be decoded by 
moving image decoding apparatus in an embodiment of the 
present invention. The moving image encoding apparatus 
200 independently encodes frames that form a moving 

25 image by an encoding method as a combination of wavelet 
transformation and bitplane encoding. As shown in 
Fig. 1, the apparatus 200 comprises a moving image data 
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input unit 201^ discrete wavelet transformer 202, 
coefficient quantizer 203, bitplane encoder 204, code 
sequence forming unit 205, secondary storage device 206, 
and signal line 207, 
5 The flow of the encoding process in the moving 

image encoding apparatus 200 will be explained while 
explaining the operations of the building components of 
the moving image encoding apparatus 200 shown in Fig. 1. 
In the following description, assume that the moving 

10 image encoding apparatus 200 fetches and encodes 
monochrome moving image data (8 -bit luminance 
value/pixel) at a rate of 30 frames /sec for four 
seconds (a total of 120 frames). That is, the moving 
image encoding apparatus 200 encodes moving image data 

15 at the rate of 30 frames per second input from the 

moving image data input unit 201 for respective frames, 
and finally stores encoded data in the secondary 
storage device 206. 

The moving image data input unit 201 inputs 

20 moving image data for four seconds at a rate of 30 

frames/sec. The moving image data input unit 201 is, 
for excmiple, an image sensing unit of a digital camera 
or the like, and can be implemented by an image sensing 
device such as a CCD or the like, and various image 

25 adjustment circuits such as a geurana correction circuit, 
shading correction circuit, and the like. The moving 
image data input unit 201 sends the input moving image 
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data to the discrete wavelet transformer 202 frame by 
frame. In tlie following description, numbers are 
assigned to frame data in turn from 1 in the order they 
are input, and respective frames are identified by 
numbers like frame 1, frame 2,.... Also, x represents 
the horizontal pixel position (coordinate) in each 
frame, y represents the vertical pixel position, and 
P(x, y) represents a pixel value at the pixel position 
(x, y) • 

Image data for one frame input from the moving 
image data input unit 201 is stored in the internal 
buffer (not shown) of the discrete wavelet transformer 
202 as needed, and undergoes 2D discrete wavelet 
transformation. The 2D discrete wavelet transformation 
is implemented by applying linear discrete wavelet 
transformation in the horizontal and vertical 
directions. Figs. 2A to 2C are views for explaining 
subbands of an image to be encoded, which is processed 
by the 2D discrete wavelet transformation. 

More specifically, linear discrete wavelet 
transformation is applied to an image to be encoded 
shown in Fig. 2A in the vertical direction to decompose 
it into low- and high-frequency subbands L and H, as 
shown in Fig. 2B. Then, linear discrete wavelet 
transformation is applied to these subbands in the 
horizontal direction to decompose them into four 
subbands LL, HL, LH, and HH, as shown in Fig. 2C. 
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The discrete wavelet transformer 202 of the 
moving image encoding apparatus 202 further applies 2D 
discrete wavelet transformation to subband LL obtained 
by the aforementioned 2D discrete wavelet 
5 transformation . As a result , the image to be encoded 
can be broken up into seven subbands LL, LHl, HLl, HHl, 
LH2, HL2, and HH2 • Fig. 3 shows the seven subbands 
obtained by the two 2D discrete wavelet transformation 
processes . 

10 In the moving image encoding apparatus 200, C(Sb, 

X, y) represents a coefficient in each subband. Note 
that Sb represents the type of subbands, i.e., one of 
LL, LHl, HLl, HHl, LH2 , HL2 , and HH2 . Also, (x, y) 
represents the coefficient position (coordinates) in 

15 the horizontal and vertical directions when the 

coefficient position of the upper left corner in each 
subband is specified by ( 0 , 0 ) . 

The moving image encoding apparatus 200 comprises 
two linear discrete wavelet transformation methods for 

20 N linear signals s(n) (n is an integer ranging from 0 

to N - 1) in the discrete wavelet transformer 202. One 
method is transformation using an integer type 5x3 
filter described by: 

h(n) = s(2n+l)-floor{(s(2n)+s(2n+2)/2) ...(1) 

25 l(n) = s(2n)+floor{(h{n-l)+h(n)+2)/4} ...(2) 

where h(n) is a coefficient of the high-frequency 
subband, l(n) is a coefficient of the low- frequency 
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subband, and floor{R> is a meixlinuin integer which is 
equal to or smaller than real number R. 

.The other method is transformation using a real number 
type 5x3 filter described by: 
5 h(n) = s(2n+l)-(s(2n)+s(2n+2) )/2 ...(3) 

l(n) = s(2n)+(h(n-l)+h(n) )/4 ...(4) 
Note that values s(n) at two ends (n < 0 and n > N - 1) 
of linear signals s(n) required to calculate equations 
(1), (2), (3), and (4) are calculated from those of 

10 linear signals s(n) (n=OtoN-l)bya known method. 

Which of the integer type 5x3 filter and the 
real number type 5x3 filter is to be applied can be 
designated for each frame by a filter selection signal 
input an external device via the signal line 207, For 

15 example, if the filter selection signal input via the 

signal line 207 is "0", the frame of interest is broken 
up using the integer type 5x3 filter; if the filter 
selection signal is "1", the frame of interest is 
broken up using the real number type 5x3 filter. 

20 The coefficient quantizer 203 quantizes 

coefficients C(Sb, x, y) of respective subbands 
generated by the discrete wavelet transformer 202 using 
quantization steps delta (Sb) determined for respective 
subbands. Let Q(Sb, x, y) be the quantized coefficient 

25 value. Then, the quantization process to be done by 
the coefficient quantizer 203 is described by: 
Q(Sb, X, y) - sign{C{Sb, x, y)} 
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xfloor{ |c(Sb, X, y) |/delta(Sb)} .-.(5) 
wliere sign{l} is a function that represents the 
positive/negative sign of integer 1, and returns 1 if 1 
is positive or -1 if 1 is negative. Also, floor{R} is 
5 a maximum integer which is equal to or smaller than 
real number R. Note that the aforementioned 
quantization process is applied only when the discrete 
wavelet transformer 202 selects and uses the real 
number type 5x3 filter. When the integer type 5x3 

10 filter is selected by the filter selection signal input 
via the signal line 207, coefficients C(Sb, x, y) are 
output as quantized coefficient values. That is, in 
this case, Q (Sb, x, y) = C(Sb, x, y) . 

The bitplane encoder 204 encodes the quantized 

15 coefficient values Q(Sb, x, y) quantized by the 

coefficient quantizer 203 . Note that various encoding 
methods such as a method of segmenting the coefficients 
of respective subbands into blocks , and independently 
encoding blocks to allow easy random access , and the 

20 like have been proposed. However, in this embodiment, 
the coefficients are encoded for respective subbands 
for the sake of simplicity. 

The quantized coefficient values Q(Sb, x, y) (to 
be simply referred to as "coefficient values" 

25 hereinafter) of respective subbands are encoded by 
expressing the absolute values of the coefficient 
values Q(Sb, x, y) in each subband by natural binary 

- 18 - 



CFM03540US / P204-0121US 



values, and applying binary arithmetic encoding to them 
from the upper to lower digits in favor of a bitplane 
direction. In the following description, Qn(x, y) 
represents a bit at the n-th digit from the LSB side 
5 when the coefficient value Q(Sb, x, y) of each subband 
is expressed by a natural binary value. Note that 
variable n that represents the digit of a binary value 
will be referred to as a bitplane number, and bitplane 
number n has the LSB (least significant bit) as the 0th 

10 digit position. 

Fig. 4 is a flow chart for explaining the 
processing sequence for encoding subbands Sb by the 
bitplane encoder 204. As shown in Fig. 4, the absolute 
values of the coefficients in subband Sb to be encoded 

15 are checked to obtain a maximum value Mabs(Sb) (step 
S601). Next, the number of digits (to be referred to 
as a digit number hereinafter) NBp(Sb) required to 
express Mabs(Sb) by a binary value upon expressing the 
absolute values of the coefficients in subband Sb is 

20 calculated (step S602) by: 

NBp(Sb) = ceil{log2(Mabs(Sb)+l)} ...(6) 
where ceil{R} is a minimum integer equal to or larger 
than real number R. 

The effective digit number NBp(Sb) is substituted 

25 in bitplane number n (step S603). Then, 1 is 

subtracted from bitplane number n to calculate n - 1, 
and n - 1 is substituted in n (step S604). 
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Furthermore, a bltplane at the n-th digit 
position is encoded using a binary arithmetic code 
(step S605), Upon encoding bits in a bitplane, bits 
are categorized into some contexts based on already 
5 encoded Information, and are encoded using different 

occurrence probability prediction models for respective 
contexts . The moving image encoding apparatus 200 uses 
MQ-Coder as the arithmetic code. Since a sequence for 
encoding a binary symbol generated in a given context 

10 using this MQ-Coder, or the initialization and 

termination sequences for the arithmetic encoding 
process have been explained in detail in ISO/ IEC154 44-1 
recommendation as the international standards of still 
images, a detailed description thereof will be omitted, 

15 In the moving image encoding apparatus 200, an 

arithmetic encoder is initialized at the beginning of 
encoding of each bitplane, and undergoes a termination 
process at the end of the process. Immediately after 
"1" to be encoded first of each individual coefficient, 

20 the sign of that coefficient is expressed by 0 or 1, 

and is arithmetically encoded. If the sign is positive, 
the arithmetic code is 0; if it is negative, the code 
is 1. For example, if the coefficient is -5 and the 
effective digit number NBp(Sb) of subband Sb to which 

25 this coefficient belongs is 6, the absolute value of 

that coefficient is expressed by a binary value 000101, 
and is encoded from the upper to lower digits upon 

- 20 - 



CFM03540US / P204-0121US 



encoding respective bltplanes . Upon encoding the 
second bltplane (In this case, the fourth digit from 
the MSB side), first "1" is encoded, and immediately 
after this encoding, the sign is arithmetically encoded. 
5 It is checked if bltplane number n = 0 (step 

S606). As a result, if n = 0 , i.e., if the LSB plane 
has been encoded in step S605 (YES in step S606), the 
encoding process of subband Sb ends. Otherwise (NO in 
step S606), the flow returns to step S604. 

10 With the above process, all coefficients of 

subband Sb are encoded to generate code sequences CS(Sb, 
n) corresponding to respective bltplanes n. The 
generated code sequences CS(Sb, n) are sent to the code 
sequence forming unit 205, and are temporarily stored 

15 in the internal buffer (not shown) of the code sequence 
forming unit 205. 

If the coefficients of all subbands have been 
encoded by the bltplane encoder 204, and all code 
sequences are stored in the internal buffer of the code 

20 sequence forming unit 205, the code sequence forming 
unit 205 reads out the code sequences stored in its 
internal memory in a predetermined order. The unit 205 
Inserts required additional information to form a code 
sequence corresponding to one frame, and outputs that 

25 code sequence to the secondary storage device 206 . 

The final code sequence generated by the code 
sequence forming unit 205 includes a header and encoded 
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data hierarchized in three levels 0 , 1 , and 2 . The 
header stores additional information required upon 
decoding such as the numbers of pixels of an image in 
the horizontal and vertical directions, the number of 
5 times of application of 2D discrete wavelet 

transf oarmation , information that designates the 
selected filter, quantization steps delta(Sb) of 
respective subbands, and the like. 

The encoded data of level 0 includes code 

10 sequences CS(LL, Nbp(LL)-I) to CS{LL, 0) obtained by- 
encoding the coefficients of LL subband. The encoded 
data of level 1 includes code sequences CS(LH1, 
Nbp{LH1)-1) to CS(LH1, 0), CS(HL1, Nbp(HLI)-I) to CS(HL1, 
0), and CS(HH1, Nbp(HHI)-I) to CS{HH1, 0) obtained by 

15 encoding the coefficients of LHl, HLl, and HHl subbands. 
The encoded data of level 2 includes code sequences 
CS(LH2, Nbp(LH2)-1) to CS(LH2, 0), CS(HL2, Nbp(HL2)-1) 
to CS(HL2, 0), and CS(HH2, Nbp(HH2)-1) to CS(HH2, 0) 
obtained by encoding the coefficients of LH2 , HL2, and 

20 HH2 subbands . 

Fig. 5 shows the detailed structure of the code 
sequence corresponding to one freime data, which is 
generated by the code sequence forming unit 205. 

The code sequence with the configuration shown in 

25 Fig. 5 allows the decoding side to obtain a 

reconstructed image of a resolution 1/4 that of the 
original image by decoding the header and encoded data 
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of level 0. Also, the decoding side can obtain a 
reconstructed Image of a resolution 1/2 that of the 
original Image by additionally decoding the encoded 
data of level 1. Furthermore, when the encoded data of 
5 level 2 Is additionally decoded, a reconstructed Image 
with the original resolution can be obtained. In this 
way, an Image can be reclaimed by gradually Increasing 
the resolution. 

On the other hand, when only some of upper 

10 bltplanes of bltplane encoded data of respective levels 
are decoded, a coarse decoded Image Is obtained. When 
the number of bltplanes to be decoded Is Increased 
toward lower bltplanes, the transform coefficients of 
respective subbands can be reconstructed while 

15 gradually Increasing the precision. Hence, the decoded 
Image quality can be Improved. 

The secondary storage device 206 comprises, e.g., 
a hard disk, memory, or the like, and Internally stores 
the code sequences generated by the code sequence 

20 forming unit 205. In the secondary storage device 206, 
the code sequences of respective frames output from the 
code sequence forming unit 205 are coupled to form 
encoded data of moving image data. Fig. 6 shows the 
structure of encoded moving image data stored in the 

25 secondary storage device 206. A header at the head of 
the sequence stores additional information for a moving 
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Image such as the number of frames, playback frame rate, 
and the like. 
<First Embodiment > 

Fig. 7 is a block diagram showing the arrangement 
5 of a moving image decoding apparatus 100 according to 
the first embodiment of the present invention. The 
same reference numerals in Fig. 7 denote blocks common 
to those in the aforementioned moving image encoding 
apparatus 200. As shown in Fig. 7, the moving image 

10 decoding apparatus 100 according to the first 

embodiment comprises a secondary storage device 206, 
code sequence read unit 101, bitplane decoder 102, 
inverse discrete wavelet transformer 104, moving image 
data output unit 105, decoding process time measurement 

15 unit 106, and non-decoding bitplane determination unit 
107. 

The operation sequence of the moving image 
decoding apparatus 100 of the first embodiment will be 
described below with reference to Fig. 7. 

20 Encoded moving image data to be decoded by the 

moving image decoding apparatus 100 of the first 
embodiment is encoded data generated by the 
aforementioned moving image encoding apparatus 200. 
Assume that the integer type 5x3 filter is used for 

25 all the frames upon generation of encoded moving image 
data. That is, in the aforementioned moving image 
encoding apparatus 200 a signal for selecting the 
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integer type 5x3 filter is input via the signal line 
207 to encode moving image data. 

Encoded moving image data is decoded for 
respective frames in encoded data. The code sequence 
5 read unit 101 reads out encoded data of a frame of 

interest from the encoded data stored in the secondary 
storage device 206, and stores it in its internal 
buffer (not shown). The encoded data for respective 
frames are read out in turn like frame 1, freune 2,..., 

10 The bit plane decoder 102 reads out the encoded 

data stored in the internal buffer of the code sequence 
read unit 101 in the order of subbands, and decodes it 
to obtain quantized transform coefficient data Q(Sb, x, 
y) . The process in the bitplane decoder 102 forms a 

15 counterpart of that in the bitplane encoder 204 shown 
in Fig. 1. 

That is, the bitplane encoder 204 encodes 
respective bits of the absolute values of coefficients 
in turn from the upper to lower bit planes using a 

20 predetermined context by binary arithmetic coding. By 
contrast, the bitplane decoder 102 decodes binary 
arithmetic encoded data using the same context as that 
upon encoding from the upper to lower bitplanes to 
decode respective bits of coefficients. As for the 

25 positive/negative sign of each coefficient, an 

arithmetic code is decoded using the same context at 
the same timing as those upon encoding. 



- 25 - 



CFM03540US / P204-0121US 

At this time, the non-decoding bitplane 
determination unit 107 designates the number ND(Sb) of 
lower bitplanes which are not to be decoded ( to be 
referred to as non-decoding bitplane number ND(Sb) 
5 hereinafter) for each subband, and the bitplane decoder 
102 skips decoding processes of the designated number 
of lower bitplanes. For example, if the non- decoding 
bitplane number ND(HH2) of subband HH2 designated by 
the non-decoding bitplane determination unit 107 is 2, 

10 the decoder 102 decodes CS(HH2, Nbp(HH2)-1) to CS(HH2, 
2) of the encoded data of the coefficients of subband 
HH2 read out from the code sequence read unit 101 to 
reclaim the coefficients of that subband, and skips 
decoding of two bitplanes CS(HH2, 1) and CS(HH2, 0). 

15 The inverse discrete wavelet transformer 104 

executes inverse transformation of the wavelet 
transformation process in the discrete wavelet 
transformer 202 in Fig. 1 to reclaim data of the frame - 
In the moving image decoding apparatus 100 of the first 

20 embodiment, since encoded moving image data generated 
using the integer type 5x3 filter for all the frames 
is to be decoded, inverse transformation corresponding 
to equations { 1 ) and ( 2 ) above is done . 

Upon playing back and displaying a moving image, 

25 respective frame data are displayed at predetermined 
timings. On the other hand, since the output timings 
from the inverse discrete wavelet transformer 104 
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depend on a time required for the decoding process, 
they are not synchronized with the display timings. 
For this reason, decoded frame data must be stored In a 
buffer to adjust a difference from Its display timing. 
5 For excunple, the moving Image data output unit 105 may 
be Implemented by a network Interface, and a buffer 
storage process for adjusting the difference from the 
display timing may be executed outside the moving image 
decoding apparatus 100 of the first embodiment, or may 

10 be executed inside the moving image output unit 105. 

As a practical exeimple of the moving image data 
output unit 105, a case will be described below wherein 
a display is connected to the moving image data output 
unit 105 to display a moving image. Fig. 16 is a block 

15 diagram showing an embodiment of the moving image data 
output unit 105 and connection to a display. Referring 
to Fig. 16, reference numeral 1601 denotes a buffer 
which can store a plurality of frame data; 1602, a 
display Interface; and 1603, a display. The buffer 

20 1601 stores reconstructed image data output from the 
inverse discrete wavelet transformer 104 at variable 
time intervals in turn. Upon storing the reconstructed 
image data, the frame number and storage address of the 
reconstructed image data are held, so that the stored 

25 data can be read out in turn later. The display 

Interface 1602 reads out the reconstructed image data 
from the buffer 1601 at given time intervals according 
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to the frame rate of moving Image data (at 1/30 -sec 
inteirvals if a moving image has 30 frames /sec) , and 
displays them on the display 1603. The readout frame 
data is erased from the buffer 1601. In this manner, 
5 the buffer 1601 serves to adjust the differences 

between the input timings of reconstructed image data 
from the inverse discrete wavelet transformer 104 and 
data read timings by the display interface. 

The moving image decoding apparatus of this 

10 embodiment executes control so that the average of 
decoding process times per frame becomes a target 
decoding time T by a process to be described later. 
However, since the times required to decode respective 
data vary, the number of frame data stored in the 

15 buffer 1601 changes. For this reason, the moving image 
playback start timing is adjusted, e.g., to start the 
playback and display processes a predetermined period 
of time after the beginning of moving image decoding or 
to start the playback and display processes after a 

20 predetermined number of frames are stored in the buffer, 
so as to avoid a situation in that given frame data is 
not prepared in the buffer when its display timing is 
reached. When the buffer size is limited, and a 
predetermined number or more of frame data are stored 

25 in the buffer, control for pausing the frame data 
decoding process is also required. 
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The moving Image data output unit 105 outputs 
reconstructed image data output from the inverse 
discrete wavelet transformer 104 to a device outside 
the apparatus. The moving image data output unit 105 
5 can be implemented by an interface to, e.g., a network 
line or display device. 

The decoding process time measurement unit 106 
measures, for each frame, a time Dt required from the 
beginning of read of encoded frame data by the code 

10 sequence read unit 101 until the moving image data 

output unit 105 outputs reconstructed frame data, and 
outputs it to the non-decoding bitplane determination 
unit 107. However, when the moving image data output 
unit 105 comprises the internal buffer that stores 

15 frame data, and adjusts the output timings to a device 
outside the apparatus to constant time intervals, as 
exemplified in Fig. 16, the time Dt is that until frame 
data is stored in the buffer 1601. 

The non-decoding bitplane determination unit 107 

20 determines non- decoding bitplanes of each subband on 

the basis of the decoding process time per frame output 
from the decoding process time measurement unit 106. 
The non-decoding bitplane determination unit 107 holds 
variables Q (to be referred to as "Q factors" 

25 hereinafter) serving as index values upon determining 
the numbers of non-decoding bitplanes, a table 
indicating the non -decoding bitplane niimbers of 
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respective subbands In correspondence with Q factors, a 
target decoding process time T, and a time difference 
AT. Fig- 8 shows an example of the table that 
represents the correspondence between the Q factors and 
5 the non-decoding bltplane numbers of respective 
subbands • 

Fig. 9 Is a flow chart showing the flow of the 
encoded moving Image data decoding process by the 
moving Image decoding apparatus 100* As shown In 

10 Fig. 9, the Q factor and time difference AT are reset 
to zero before the decoding start timing of encoded 
moving Image data. I.e., before the beginning of 
decoding of encoded data of frame 1 (step S701). 

The non-decoding bltplane determination unit 107 

15 reads out the non-decoding bltplane number of each 
subband from the table on the basis of the Q factor, 
and sets It In the bltplane decoder 102 (step S702). 

Next , one frame Is decoded by the processes from 
the code sequence read unit 101 to the Inverse discrete 

20 wavelet transformer 104, and frame data Is output to 
the moving Image data output unit 105 (step S703). 

The decoding process time measurement unit 106 
measures a time Dt required for the decoding process 
for one freime executed In step S703, and passes It to 

25 the non-decoding bltplane determination unit 107 (step 
S704). 
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The non- decoding bitplane determination unit 107 
calculates a difference between the target decoding 
time T for one frame and the actually required decoding 
process time Dt, and adds it to the held time 
5 difference AT (step S705). 

Next, the Q factor is updated according to the 
value of AT (step S706). If AT is larger than a 
predetermined threshold value Uq (Uq > 0) which is set 
in advance, 1 is subtracted from Q to decrease the 

10 value Q. AT becomes larger than the predetermined 
threshold value when the sum total of the actually 
required decoding times is smaller than that of the 
target times. Hence, in order to improve the decoded 
image quality, the value Q is decreased to decrease the 

15 non-decoding bitplane number. By contrast, if AT is 
smaller than a predetermined threshold value Lq (Lq < 
0) which is set in advance, 1 is added to Q tb increase 
the value Q. AT becomes smaller than the predetermined 
threshold value Lq when the sum total of the actually 

20 required decoding times is larger than that of the 

target times. Hence, in order to shorten a decoding 
time per frame, the value Q is increased to increase 
the non-decoding bitplane number. Note that the value 
Q ranges from 0 to 9. If Q < 0 after the above update 

25 process, Q = 0;ifQ>9, Q==9. When AT falls within 
the range between the threshold values Lq and Uq (Lq ^ 
AT ^ Uq), since the sum total of the actually required 
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decoding times falls within an optimal range with 
respect to that of the target times, the value Q 
remains unchanged. 

It is checked if the frame that has undergone the 
5 decoding process is the last frame. If the frame of 
interest is not the last frame (NO in step S707), the 
flow returns to step S702 to decode the next frame; 
otherwise (YES in step S707), the decoding process of 
the encoded moving image data ends (step S707). 

10 As described above, the Q factor as an index 

value of the non- decoding bitplane number is determined 
based on the accumulated value of the differences each 
between the time required for the decoding process per 
frame and the target decoding time, and the 

15 non -decoding bitplane number is changed in accordance 
with the Q factor, thus controlling the decoding 
process time while suppressing visual problems of a 
playback image as much as possible. 
< Second Embodiment > 

20 The first embodiment has been described under the 

condition that subband decomposition is made using an 
integer type 5x3 filter for all frames of encoded 
moving image data to be decoded. In a moving image 
decoding apparatus of the second embodiment , a case 

25 will be explained wherein encoded moving image data 
which is generated by subband decomposition using a 
real number type 5 x 3 filter is to be decoded. That 
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is, in the aforementioned moving image encoding 
apparatus 200 a signal for selecting the real niimber 
type 5x3 filter is input via the signal line 207 to 
encode moving image data. Upon encoding, all frames 
5 use an identical quantization step delta (Sb) of each 
subband . 

Fig. 10 is a block diagram showing the 
arrangement of a moving image decoding apparatus 300 
according to the second embodiment of the present 

10 invention. The Scune reference numerals in Fig. 10 
denote blocks common to those in the aforementioned 
moving image encoding apparatus 200 and the moving 
image decoding apparatus 100 of the first embodiment . 
As shovm in Fig. 10, the moving image decoding 

15 apparatus 300 according to the second embodiment 

comprises a secondary storage device 206, code sequence 
read unit 904, bitplane decoder 102, coefficient 
dequantizer 901, inverse discrete wavelet transformer 
902, moving image data output unit 105, decoding 

20 process time measurement unit 106, and non-decoding 
bitplane determination unit 903. 

The operation sequence of the moving image 
decoding apparatus 300 of the second embodiment will be 
described below using Fig. 10. 

25 The code sequence read unit 904 reads out encoded 

data of a frame of interest from the encoded data 
stored in the secondary storage device 206, and stores 
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it in its internal buffer (not shown) as in the code 
sequence read unit 101 of the first embodiment. At 
this time, the unit 904 reads out a quantization step 
delta (Sb) of each subband Sb from the header of the 
5 readout encoded data of the frame, and stores it in the 
internal buffer (not shown) . 

The bitplane decoder 102 decodes the encoded data 
stored in the internal buffer of the code sequence read 
unit 904 to obtain quantized transform coefficient data 

10 Q(Sb, X, y) , as in the first embodiment. In the moving 
image decoding apparatus 300 of the second embodiment 
as well, the decoder 102 skips decoding processes of 
ND(Sb) lower bitplanes designated by the non-decoding 
bitplane determination unit 903. 

15 The coefficient dequantizer 901 reclaims 

coefficients C(Sb, x, y) of each subband based on the 
quantization step delta(Sb) determined for each subband 
and the quantized coefficient data Q(Sb, x, y) decoded 
by the bitplane decoder 102. 

20 The inverse discrete wavelet transformer 902 

executes inverse transformation of the wavelet 
transformation process in the discrete wavelet 
transformer 202 in Fig. 1 to reclaim data of the frame. 
In the moving image decoding apparatus 300 of the 

25 second embodiment, since encoded moving image data 

generated using the real number type 5x3 filter for 
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all the frames is to be decoded, inverse transformation 
corresponding to equations (3) and (4) above is done. 

The moving image data output unit 105 outputs 
reconstructed image data output from the inverse 
5 discrete wavelet transformer 902 to a device outside 
the apparatus . 

The decoding process time measurement unit 106 
measures, for each frame, a time Dt required from the 
beginning of read of encoded frame data by the code 

10 sequence read unit 904 until the moving image data 

output unit 105 outputs reconstructed frame data, and 
outputs it to the non- decoding bitplane determination 
unit 903, as in the first embodiment. 

The non-decoding bitplane determination unit 903 

15 determines non-decoding bitplanes of each subband on 

the basis of the decoding process time per fremie output 
from the decoding process time measurement unit 106. 
The non-decoding bitplane determination unit 903 holds 
a table indicating the non-decoding bitplane numbers 

20 ND(Sb) of respective subbands in correspondence with Q 
factors, a target decoding process time T, a time 
difference AT, and a subband index SI. Fig. 11 shows 
an example of the table that holds the non -decoding 
bitplane numbers ND(Sb) for respective subbands Sb. 

25 Fig. 12 is a flow chart showing the flow of the 

encoded moving image data decoding process by the image 
decoding apparatus 300 of this embodiment. As shown in 
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Fig. 12, the subband Index SI and time difference AT 
are reset to zero before the decoding start timing of 
encoded moving image data, i.e., before the beginning 
of decoding of encoded data of frame 1 (step SllOl). 
5 All non- decoding bit plane numbers ND(Sb) for 

respective subbands held in the non- decoding bitplane 
determination unit 903 are reset to zero (step S1102). 

The non-decoding bitplane numbers ND(Sb) stored 
in the non -decoding bitplane determination unit 903 are 
10 read out and are set in the bitplane decoder 102 (step 
S1103) . 

Next, one freune is decoded by the processes from 
the code sequence read unit 904 to the inverse discrete 
wavelet transformer 902, and frame data is output to 
15 the moving image data output unit 105 (step S1104). 

The decoding process time measurement unit 106 
measures a time Dt required for the decoding process 
for one frame executed in step S1104, and passes it to 
the non-decoding bitplane determination unit 903 (step 
20 S1105). 

The non-decoding bitplane determination unit 903 
calculates a difference between the target decoding 
time T for one frame and the actually required decoding 
process time Dt, and adds it to the held time 
25 difference AT (step S1106). 
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The table that holds the non- decoding bit plane 
numbers ND(Sb) and subband index SI are updated 
according to the value of AT (step S1107). 

Fig. 13 is a flow chart showing the flow of the 
5 process executed in step S1107. It is checked if AT is 
larger than a predetermined threshold value Uq (Uq > 0) 
which is set in advance (step S1201). If AT > Uq (YES 
in step S1201), the subband index SI is decremented by 
1 (step S1202). It is then checked if SI = -1 (step 

10 S1203). If SI = -1, SI is set to be 6 (step S1204). 
Then, 1 is subtracted from the non-decoding bitplane 
number NS(S(SI)) of subband S(SI) corresponding to the 
subband index SI (step S1205). AT becomes larger than 
the predetermined threshold value when the sum total of 

15 the actually required decoding times is smaller than 
that of the target times. Hence, the decoding image 
quality is improved by decreasing the non-decoding 
bitplane number. The correspondence between the 
subband indices SI and subbands is as shown in Fig. 14. 

20 For example, if SI = 2, the corresponding subband is 
HL2, and 1 is subtracted from the value of ND(HL2). 
Then, ND(S(SI)) is compared with 0 (step S1206). If 
ND(S(SI)) assumes a value smaller than 0, ND(S(SI)) is 
set to be zero (step S1207), and SI is reset to zero 

25 (step S1213) . 

On the other hand, if AT ^ Uq as a result of 
comparison in step S1201 (i.e., NO in step S1201), AT 
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is compared with a predetermined threshold value Lq (Lq 
< 0) which is set in advance (step S1208). If AT > Lq 
(NO in step S1208), the process ends. If AT ^ Lq (YES 
in step S1208), ND(S(SI)) is incremented by 1 (step 
5 S1209). AT becomes smaller than the predetermined 
threshold value when the siun total of the actually 
required decoding times is larger than that of the 
target times. Hence, the decoding time per frame is 
shortened by increasing the non-decoding bitplane 

10 number. Next, SI is also incremented by 1 (step S1210), 
and is compared with 7 (step S1211). If SI = 7 (YES in 
step S1211), SI is set to be zero (step S1212). 

With the above process, if AT is larger than the 
predetermined value or is smaller than another 

15 predetermined value, the non-decoding bitplane number 
ND(Sb) of one subband is changed by one level. 

The description will revert to the process in 
Fig. 12. It is checked if the frame that has undergone 
the decoding process is the last frame (step S1108). 

20 If the frame of interest is not the last frame (NO in 
step S1108), the flow returns to step S1103 to decode 
the next frame; otherwise (YES in step S1108), the 
decoding process of the encoded moving image data ends. 
As described above, the non-decoding bitplane 

25 number for each subband is changed in accordance with 
the accumulated value of the differences each between 
the time required for the decoding process per frame 
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and the target decoding time, thus controlling the 
decoding process time while suppressing visual problems 
of a playback image as much as possible. 
<Third Embodiment > 
5 In the moving image decoding apparatuses of the 

first and second embodiments, a non-decoding part is 
determined for respective bitplanes . Alternatively, 
bits in a bitplane may be categorized into a plurality 
of passes (sub-bitplanes ) based on encoded parts around 

10 the bit of interest, and a non-decoding part may be 

determined for respective passes. An embodiment that 
determines the non-decoding parts for respective passes 
will be explained below. 

The process for generating encoded moving image 

15 data to be decoded by the moving image decoding 

apparatus of the third embodiment is basically the same 
as that of the moving image encoding apparatus 200 
shown in Fig. 1, except for the method of encoding 
bitplanes in the bitplane encoder 204, i.e., that the 

20 encoder 204 encodes one bitplane while breaking it up 
into a plurality of passes, as described above. 
Although a description of a detailed pass segmentation 
method will be omitted for the sake of simplicity, 
encoding is done by the same method as the bitplane 

25 encoding method in JPEG2000 described in the 

ISO/IEC15444-1 recommendation. In JPEG2000, encoding 
is done while breaking up into three passes except for 
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the most significant bitplane. Therefore, if the 
effective bit number of given subband Sb is NBp(Sb), 
encoding is done by (NBp(Sb) - 1) x 3 + 1 passes. Let 
CSP(Sb, n) be codes generated by respective passes (n 
5 is a pass number; the first pass number = (NBp(Sb) - 1) 
X 3) and the last pass number = 0). 

The code sequence forming unit 205 forms a code 
sequence by arranging encoded data of passes in the 
same manner as in the above case wherein it forms a 

10 code sequence by arranging encoded data for respective 
bitplanes. Fig. 15 shows an example of the structure 
of encoded moving image data which is generated in this 
way, and is to be decoded by the moving image decoding 
apparatus of the third embodiment. A difference from 

15 the data to be decoded of the first and second 

embodiment shown in Fig. 5 is that pass encoded data 
CSP{Sb, n) replace bitplane encoded data CS{Sb, n) as 
elements which form encoded data (n is a bitplane or 
pass number). Furthermore, the encoded moving image 

20 data to be decoded in the first and second embodiments 
include all bitplane encoded data output from the 
bitplane encoder 204. However, in this embodiment, the 
code sequence forming unit 205 discards some encoded 
data. That is, the unit 205 discards encoded data of 

25 the last passes for subbands HHl, LH2 , and HL2, and 
discards encoded data of the last two passes for HH2. 
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Note that the arrangement of the moving Image 
decoding apparatus of the third embodiment is the same 
as that of the moving image decoding apparatus 100 of 
the first embodiment shown in Fig, 7, and only the 
5 operations of the bitplane decoder 102 and non-decoding 
bitplane determination unit 107 are different from the 
first embodiment. 

The bitplane decoder 102 extracts bits of 
respective passes by decoding pass encoded data CSP(Sb, 

10 n) by the decoding process that forms a counterpart of 
the bitplane encoding process of the aforementioned 
moving image encoding apparatus 200, thus reclaiming 
subband coefficients. At this time, in the moving 
image decoding apparatus 100 of the first embodiment, 

15 the non-decoding bitplane determination unit 107 

designates the lower bitplane numbers ND(Sb) which are 
not to be decoded. However, in the third embodiment, 
the non-decoding bitplane determination unit 107 
designates the numbers NDP(Sb) of lower passes which 

20 are not to be decoded, and the bitplane decoder 102 
skips the decoding process of lower NDP(Sb) passes. 
More specifically, the decoder 102 skips the decoding 
process from CSP(Sb, NDP(Sb) - 1) to CSP(Sb, 0). 

The non- decoding bitplane determination unit 107 

25 designates the non-decoding pass numbers NDP(Sb) by the 
same process as that for designating the non-decoding 
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bltplane numbers ND(Sb) to the bitplane decoder 102 in 
the first embodiment. 

As described above, since the moving image 
decoding apparatus 100 of the third embodiment can set 
5 a non- decoding part using smaller encoding units than 
bitplanes, the decoded image quality and decoding time 
can be adjusted more finely. 
< Fourth Embodiment > 

In the first to third embodiments, the decoding 

10 range is adjusted based on the accumulated value AT of 
the differences each between the target decoding time T 
for each freune and actually required decoding process 
time Dt . As has been explained in the practical 
example of the moving image data output unit 105 in the 

15 first embodiment, when the moving image data output 
unit 105 comprises the buffer and display interface, 
and the display is connected to display a moving image, 
an equivalent process can be implemented by monitoring 
the number of frame data stored in the buffer without 

20 actually measuring the decoding process time. Such 
embodiment will be described below. 

The process for generating encoded moving image 
data to be decoded by the moving image decoding 
apparatus of the fourth embodiment is the same as that 

25 of the moving image encoding apparatus 200 shown in 
Fig. 1. 



- 42 - 



I 



CFM03540US / P204-0121US 

Fig. 17 is a block diagram showing the 
arrangement of a moving image decoding apparatus 400 
according to the fourth embodiment of the present 
invention. The same reference numerals in Fig. 17 
5 denote blocks common to those in the aforementioned 
moving image encoding apparatus 200 and the moving 
image decoding apparatus 100 of the first embodiment. 
As shown in Fig. 17, the moving image decoding 
apparatus 400 according to the fourth embodiment 

10 comprises a secondary storage device 206, code sequence 
read unit 101, bitplane decoder 102, inverse discrete 
wavelet transformer 104, buffer 1601, display interface 
1602, non-decoding bitplane determination unit 1702, 
and buffer state monitor unit 1701, and is connected to 

15 a display 1603. As shown in Fig. 17, in the moving 
image decoding apparatus 400 according to the fourth 
embodiment of the present invention, the moving image 
data output unit 105 of the moving image decoding 
apparatus 100 of the first embodiment comprises the 

20 buffer 1601 and display interface 1602, as shown in 

Fig. 16, the decoding process time measurement unit 106 
is replaced by the buffer state monitor unit 1701, and 
the non-decoding bitplane determination unit 107 is 
replaced by the non-decoding bitplane determination 

25 unit 1702. 

The operation sequence of the moving image 
decoding apparatus 400 of the fourth embodiment will be 
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described below with reference to Fig. 17. The 
operations of the code sequence read unit 101, bltplane 
decoder 102, and Inverse discrete wavelet transformer 
104 are the same as those In the moving Image decoding 
5 apparatus 100 of the first embodiment. Also, the basic 
operations of the buffer 1601 and display Interface 
1602 are as have been explained In the practical 
example of the moving Image data output unit 105 In the 
first embodiment, which comprises a buffer and display 

10 Interface. However, assume that the moving Image 

decoding apparatus 400 of the fourth embodiment starts 
display of decoded frame data after an elapse of a time 
mT from the beginning of decoding so as to avoid a 
situation In that data to be displayed at a 

15 predetermined timing Is not prepared In the buffer 1601 
(T Is the target decoding time per frame as In the 
above embodiment, and m Is an arbitrary positive 
Integer). That Is, the display Interface 1602 reads 
out decoded frame data In turn from the buffer 1601 at 

20 predetermined time Intervals after an elapse of the 
time mT from the beginning of the decoding process, 
thus displaying them on the display. As a time 
required for preparing decoded data for one frame on 
the buffer 1601, If elements other than the decoding 

25 process time can be Ignored, In case of a moving Image 
that Includes 30 frames /sec, the target decoding time T 
Is 1/30 sec, and the time Interval upon reading out 
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decoded frame data from the buffer 1601 Is also 1/30 
sec . 

Fig. 18 chronologically shows the operations of 
the moving image decoding apparatus from the beginning 
5 of decoding. As shown in Fig. 18, a first frame is 
displayed at a timing after an elapse of the time mT 
from the decoding start timing (time 0), and frames are 
displayed in sequence at time intervals T. The time 
difference AT which is used as an index upon 

10 controlling the decoding range in the first to third 
embodiments is calculated by subtracting the time 
required to decode given frame n of interest from the 
time nT, and the control is made based on this value. 
That is, if this value is larger than the upper limit 

15 value Uq, the decoding range is broadened to improve 
the decoded image quality; if this value is smaller 
than the lower limit value Lq, the decoding range is 
narrowed to reduce the decoded image quality and to 
shorten the decoding process time. The moving image 

20 decoding apparatus of the fourth embodiment implements 
equivalent control by focusing attention on the number 
of decoded frame data stored in the buffer 1601. 

Referring to Fig. 18, let d be the time required 
from the start timing of decoding until the decoding 

25 end timing of frame n. Then, if (n-l)T < d < nT, i.e., 
if 0 < AT < T, the number of decoded frames stored in 
the buffer 1601 at the decoding end timing of frame n 
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is m. If AT > T, m+1 or more decoded freune data are 
stored in the buffer 1601 at the decoding end timing of 
frame n* Conversely, if nT < d < {n+l)T, i.e., if -T < 
AT < T, the number of decoded frames stored in the 
5 buffer 1601 at the decoding end timing of frame n is 
m-1. If AT < -T, m-2 or less frames are stored. If 
the number of decoded frames stored in the buffer 1601 
at the decoding end timing of frame n is m+1 or more, 
control is made to broaden the decoding range so as to 

10 improve the decoded image quality; if the number of 
stored freunes is m-2 or less, the control is. made to 
reduce the decoded image quality and to shorten the 
decoding process time per frame. In this manner, the 
Scune operations as those upon controlling the decoding 

15 range based on AT (Uq = T and Lq = -T) can be made. 

Assume that the decoded frame data storage process in 
the buffer 1601 by the inverse discrete wavelet 
transformer 104 and the read process of decoded frame 
data by the display interface 1602 do not take place at 

20 the S£une time. 

The buffer state monitor unit 1701 acquires the 
number p of decoded frames which are stored in the 
buffer 1601 at every decoding end timing of a frame, 
and outputs it to the non-decoding bitplane 

25 determination unit 1702. 

The non-decoding bitplane determination unit 1702 
determines non- decoding bitplanes of respective 
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subbands on the basis of the number p of stored frames 
output from the buffer state monitor unit 1701. The 
non-decoding bitplane determination unit 1702 holds 
variables Q (to be referred to as "Q factors" 
5 hereinafter) serving as index values upon determining 
the numbers of non-decoding bitplanes, a table 
indicating the non-decoding bitplane numbers of 
respective subbands in correspondence with Q factors , 
and a parameter m used to determine a delay time from 

10 the decoding start timing to the display start timing. 
Fig. 8 shows an example of the table that represents 
the correspondence between the Q factors and the 
non-decoding bitplane numbers of respective subbands. 

Fig. 19 is a flow chart showing the flow of the 

15 encoded moving image data decoding process by the 
moving image decoding apparatus 400. As shown in 
Fig. 19, the Q factor and time difference AT are reset 
to zero before the decoding start timing of encoded 
moving image data, i.e., before the beginning of 

20 decoding of encoded data of frame 1 (step S1901). 

The non-decoding bitplane determination unit 1702 
reads out the non- decoding bitplane number of each 
subband from the table on the basis of the Q factor, 
and sets it in the bitplane decoder 102 (step S1902) . 

25 Next, one freune is decoded by the processes from 

the code sequence read unit 101 to the inverse discrete 
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wavelet transformer 104, and freune data is stored in 
the buffer 1601 (step S1903). 

The buffer state monitor unit 1701 acquires the 
number p of frame data stored in the buffer 1601, and 
5 passes it to the non-decoding bitplane determination 
unit 1702 (step S1904). 

It is checked whether or not the time mT has 
elapsed from the beginning of the decoding process, 
i.e., whether or not display of decoded frame data has 

10 started (step S1905). If display has not started yet 
(NO in step S1905), the flow returns to step S1902 to 
process the next frame. If display has already started 
(YES in step S1905), the Q factor is updated according 
to the value p (step S1906). If p is larger than m 

15 (i.e. , p > m) , 1 is subtracted from Q to decrease the 
value Q. p > m when the sum total of the actually 
required decoding times is smaller than that of the 
target times . Hence , in order to improve the decoded 
image quality, the value Q is decreased to decrease the 

20 non-decoding bitplane number. Conversely, if p is 

smaller than m-1 (i.e., p < m-1), 1 is added to Q to 
increase the value, p < m-1 when the sum total of the 
actually required decoding times is larger than that of 
the target times. Hence, in order to shorten the 

25 decoding time for one frame, the value is increased to 
increase the number of non-decoding bitplanes. Note 
that the value Q ranges from 0 to 9 . If Q < 0 after 
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the above update process, Q=0;ifQ>9, Q=9. Ifp 
= m, since the sum total of the actually required 
decoding times falls within an optimal range with 
respect to that of the target times, the value Q 
5 remains unchanged. 

It is checked in step S1907 if the frcime that has 
undergone the decoding process is the last frame. If 
the frame of interest is not the last frame (NO in step 
S1907), the flow returns to step S1902 to decode the 

10 next frame; otherwise (YES in step S1907), the decoding 
process of the encoded moving image data ends. 

Even after the end of the decoding process, the 
display interface 1602 continues the read process of 
frame data stored in the buffer 1601 for the time mT. 

15 As described above, the moving image decoding 

apparatus 400 of the fourth embodiment adjusts the 
decoding range using the number of decoded frcimes held 
in the buffer as a method equivalent to that for 
adjusting the decoding range on the basis of the 

20 accumulated value of the differences each between the 
time required for the decoding process per frame and 
the target decoding time. With this arrangement, even 
when it is difficult to measure the process time of 
each individual frame, the nximber of non- decoding 

25 bitplanes is changed in accordance with the number of 

decoded frames held in the buffer, thus controlling the 



- 49 - 



CFM03540US / P204-0121US 



decoding process time while suppressing visual problems 
of a playback Image as much as possible. 
< Another Embodiment > 

The present invention is not limited to the above 
5 embodiments. For example, in the first to third 

embodiments, bitplane encoding is done for respective 
subbands. Alternatively, each subband may be segmented 
into blocks , and bitplane encoding may be made for 
respective blocks. Also, one bitplane may be encoded 

10 using a plurality of passes. 

In the above example, MQ-Coder is used as the 
binary arithmetic encoding method. However, the 
present Invention is not limited to the above 
embodiments. For example, arithmetic encoding methods 

15 other than MQ-Coder such as QM-Coder and the like may 
be adopted, and other binary encoding methods may be 
adopted as long as they are suited to encode a 
multi-context information source. 

The filters for subband decomposition are not 

20 limited to those in the above embodiments. For example 
other filters such as a real number type 9x7 filter 
and the like may be used. Furthermore, the number of 
times of application of the filter is not limited to 
that in the above embodiments . In the above 

25 embodiments, linear discrete wavelet transformation is 
applied the same number of times in the horizontal and 
vertical directions, but need not always be applied the 
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same number of times in the horizontal and vertical 
directions . 

Moreover, the structure of the encoded moving 
image data is not limited to the above embodiments. 
5 For example, the order of the code sequence, the 

storage format of additional information, and the like 
may be modified. For example, the present invention is 
suitable for use of JPEG2000 specified by 
ISO/IEC15444-1 as the frame data encoding method, and 

10 encoded data described in the JPEG2000 standard or that 
of Motion JPEG2000 specified by the JPEG2000 standard 
Part 3 may be used. 

In addition, the measurement method of the 
decoding process time is not limited to the above 

15 embodiment. For example, it may be estimated that 

wavelet transformation processes and the like require a 
nearly constant process time, and only a time required 
for bitplane decoding may be measured. Also, a process 
time for a plurality of frames may be measured to 

20 control a non-decoding part. 
<Fifth Embodiment > 

Fig. 26 shows the basic arrangement of an image 
decoding apparatus according to this embodiment. 

Reference numeral 51401 denotes a CPU which 

25 controls the overall apparatus using programs and data 
stored in a RAM 51402 and ROM 51403, and executes an 
image decoding process to be described later. 
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The RAM 51402 has an area for temporarily storing 
programs and data loaded from an external storage 
device 51407 and storage medium drive 51408 or those 
which are downloaded from an external device via an I/F 
5 51409, and also has a work area used when the CPU 51401 
executes various processes. 

The ROM 51403 stores a boot program, and a setup 
program and data of this apparatus. 

Reference numerals 51404 and 51405 denote a 
10 keyboard and mouse, which allow the user to input 
various instructions to the CPU 51401. 

Reference numeral 51406 denotes a display device 
which comprises a CRT, liquid crystal display, or the 
like, and can display information such as images, text, 
15 and the like. 

The external storage device 51407 comprises a 
large -capacity information storage device such as a 
hard disk drive or the like. The external storage 
device 51407 saves an OS, programs for an image 
20 decoding process to be described later, encoded data of 
moving images whose frames serve as images to be 
encoded, and the like. These programs and data are 
loaded onto a predetermined area on the RAM 51402 under 
the control of the CPU 51401. 
25 The storage medium drive 51408 reads out programs 

and data recorded on a storage medium such as a CD-ROM, 
DVD-ROM, or the like, and outputs them to the RAM 51402 
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and external storage device 51407. Note that the 
storage medium may record programs for an image 
decoding process to be described later, encoded data of 
moving images whose frames serve as images to be 
5 encoded, and the like. In this case, the storage 

medium drive 51408 loads these programs and data onto a 
predetermined area on the RAM 51402 under the control 
of the CPU 51401. 

The I/F 51409 connects this apparatus to an 

10 external device, and allows data communications between 
this apparatus and the external device. For example, 
the aforementioned moving image encoding apparatus may- 
be connected to the I/F 51409 to input encoded moving 
image data generated by that moving image encoding 

15 apparatus to the RAM 51402 and external storage device 
51407 of this apparatus. 

Reference numeral 51410 denotes a bus which 
interconnects the aforementioned units. 

Note that the arrangement shown in Fig. 26 may be 

20 applied to the decoding apparatuses of the above 
embodiments . 

Fig. 21 is a block diagram showing the functional 
arrangement of the image decoding apparatus according 
to this embodiment . The Scune reference numerals in 
25 Fig. 21 denote the same parts as those in Fig. 1. As 
shown in Fig. 21, the image decoding apparatus 
according to this embodiment comprises a code sequence 
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read unit 5101, bxtplane decoder 5102, inverse discrete 
wavelet transformer 5104, moving image data output unit 
5105, decoding process time measurement unit 5106, and 
non-decoding subband determination unit 5107. Note 
5 that the arrangement shown in Fig. 21 may be 

implemented by hardware. However, in this embodiment, 
the respective units shown in Fig. 21 are implemented 
by a program which makes a computer implement the 
functions of the respective units, and this program is 

10 loaded from the external storage device 51407 or 
storage medium driver 51408/ or from the external 
device via the I/F 51409 onto the RAM 51402. 

In Fig. 21, reference numeral 206 denotes a 
secondary storage device described above, from which 

15 encoded moving image data is input to this apparatus. 
Assume that this secondary storage device 206 
corresponds to the external storage device 51407 or 
storage medium driver 51408, or the external device 
connected to this apparatus via the I/F 51409, and 

20 encoded moving image data is loaded onto the RAM 51402 
from one of these devices. Note that this encoded 
moving image data is generated by the moving image 
encoding apparatus , as described above . 

The process to be executed by the image decoding 

25 apparatus according to this embodiment will be 
described below with reference to Fig. 21. 
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Encoded moving Image data to be decoded by the 
Image decoding apparatus according to this embodiment 
Is generated by the aforementioned moving image 
encoding apparatus . Upon generating this encoded 
5 moving Image data, the Integer type 5x3 filter is 
used for all the frames . That is , in the 

aforementioned moving image encoding apparatus a signal 
for selecting the Integer type 5x3 filter is input 
via the signal line 207 to encode moving image data. 

10 The image decoding apparatus according to this 

embodiment initially extracts encoded image data of an 
arbitrary frame from the encoded moving image data to 
be decoded as a sample, and measures times required for 
the decoding processes of respective subbands, thus 

15 estimating the decoding process time. 

In general, encoded moving image data is decoded 
for each frame in that data. The code sequence read 
unit 5101 shown in Fig. 21 reads out encoded data of a 
frame of interest from the encoded moving image data 

20 stored in the secondary storage device 206, and stores 
it in the RAM 51402. The encoded data for respective 
frames are basically read out in turn like frame 1 , 
frame 2 , . . . , but this embodiment reads but encoded data 
of an arbitrary frame as a sample and stores it in the 

25 RAM 51402 at the beginning of the decoding process, so 
as to estimate the decoding process time by measuring 
the times required for the decoding processes of 
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respective subbands . In the following description, 
assume that the 60th freune is used as a frame to be 
decoded to estimate the decoding process time (to be 
referred to as a sample frame hereinafter) for the sake 
5 of simplicity, but the present invention is not limited 
to such specific frame. 

The bitplane decoder 5102 reads out the encoded 
data stored in the RAM 51402 in the order of subbands, 
and decodes it to obtain quantized transform 

10 coefficient data Q(Sb, x, y) . The process in the 
bitplane decoder 5102 is opposite to that in the 
bitplane encoder 204 shown in Fig. 1. 

That is, the bitplane encoder 204 encodes 
respective bits of the absolute values of coefficients 

15 in turn from the upper to lower bitplanes using a 

predetermined context by binary arithmetic coding. By 
contrast, the bitplane decoder 5102 decodes binary 
arithmetic encoded data using the same context as that 
upon encoding from the upper to lower bitplanes to 

20 decode respective bits of coefficients. As for the 
positive/negative sign of each coefficient, an 
arithmetic code is decoded using the same context at 
the same timing as those upon encoding. 

At this time, data F(Sb) indicating whether or 

25 not to decode each subband is input from the 

non-decoding subband determination unit 5107. The 
bitplane decoder 5102 decodes respective bitplanes of 
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subband Sb whicb corresponds to F ( Sb ) = 1 , and skips 
decoding of subband Sb which corresponds to F(Sb) = 0. 
This data F(Sb) Is generated by the non-decoding 
subband determination unit 5107, and this generation 
5 process will be described later. 

Note that F(Sb) = 1 Is set for all subbands Sb of 
the Scimple frame which Is to be decoded first by the 
Image decoding apparatus of this embodiment , thus 
decoding all subbands. Assume that the data F(Sb) for 
10 the sample frame Is loaded In advance onto the RAM 
51402. 

The Inverse discrete wavelet transformer 5104 
executes Inverse transformation of the wavelet 
transformation process In the discrete wavelet 

15 transformer 202 In Fig. 1 to reclaim data of the frame. 
In this embodiment , since encoded moving Image data 
generated using the Integer type 5x3 filter for all 
the frames Is to be decoded, as described above. 
Inverse transformation corresponding to equations (1) 

20 and (2) above Is done. 

The moving Image data output unit 5105 outputs 
reconstructed Image data output from the Inverse 
discrete wavelet transformer 5104 to an external device 
via the I/F 51409 or outputs It to the display device 

25 51406. However, since data of the sample freime, which 
Is decoded first by the Image decoding apparatus of 
this embodiment , Is decoded for the purpose of 
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estimating the decoding process time, the decoded frame 
data is discarded and Is not output . 

Upon playing back and displaying a moving image, 
respective frame data are displayed at predetermined 
5 timings. On the other hand, since the output timings 
from the Inverse discrete wavelet transformer 5104 
depend on the time required for the decoding process, 
they are not synchronized with the display timings. 
For this reason, decoded freune data must be stored in a 

10 buffer to adjust a difference from its display timing. 
For example, the moving image data output unit 5105 may 
output the reconstructed image data to the external 
device via the I/F 51409 to execute a buffer storage 
process for adjusting the difference from the display 

15 timing outside the moving image decoding apparatus of 

this embodiment , or that process may be executed inside 
the moving image output unit 5105. 

As a practical exsunple of the moving image data 
output unit 5105, a display may be connected to the 

20 moving image data output unit 5105 to display a moving 
image in some cases, and the arrangement at that time 
is as shown in Fig. 16. The display 1603 corresponds 
to the display device 51406 shown in Fig. 26, and a 
connection line between the display Interface 1602 and 

25 the display device 51406 corresponds to the bus 51410. 

The decoding process time measurement unit 5106 
measures times (decoding times Dt(SI)) required to 
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decode the coefficients of respective subbands of the 
sample frame by the bitplane decoder 5102 (Si is an 
index value that specifies each subband) . That is, if 
a decoding unit of the decoding process by the bitplane 
5 decoder 5102 is "subband", the decoding process time 
measurement unit 5106 measures a decoding time Dt(SI) 
for each decoding unit. The decoding process time 
measurement unit 5106 forms a table shown in Fig. 22 
based on this measurement result, and stores this table 

10 data in the RAM 51402. 

Fig. 22 shows an example of the configuration of 
the table which represents the relationship between 
subband indices SI and coefficient decoding time 
(required decoding times) of subbands Sb. Fields Dt(0) 

15 to Dt{6) store actually measured times. By looking up 
this table, for example, the coefficient decoding time 
of subband HHl (SI = 3) can be specified as Dt(3). 

Also, the decoding process time measurement unit 
5106 measures a time Dt required from the beginning of 

20 read of encoded frame data by the code sequence read 
unit 5101 from the secondary storage device 206 until 
the moving image data output unit 5105 outputs 
reconstructed frame data as a "time required to decode 
encoded image data for one frame" , and outputs it to 

25 the non-decoding subband determination unit 5107. In 
this embodiment, the decoding process time measurement 
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unit 5106 measures the time Dt required to decode 
encoded Image data of the sample frame first. 

The decoding process time measurement unit 5106 
measures the time Dt for each of frames other than the 
5 first frame in the moving image, and outputs it to the 
non-decoding subband determination unit 5107, 

The non- decoding subband determination unit 5107 
executes a process for determining a subband which is 
not to be decoded of those which form the encoded image 

10 data for one frame, so that the decoding process time 

Dt of the saunple frame output from the decoding process 
time measurement unit 5106 becomes equal to or smaller 
than a value pre- stored in the RAM 51402 as an upper 
limit value of the time required to reclaim and output 

15 image data for one frame. More specifically, the unit 
5107 determines whether or not each subband is to be 
decoded . 

Assume that the RAM 51402 holds, before this 
process, a variable M indicating the number of 

20 non-decoding subbands, data of a decoding flag F(Sb) 

(in an initial state) indicating if each subband is to 
be decoded, a variable T indicating the upper limit 
(target decoding process time) of the time required to 
reclaim and output image data for one frame, and a 

25 variable AT indicating the time difference. 

A subband (or subbands) which is not to be 
decoded is determined by a process to be described 
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later, and data of a decoding flag (or flags) F(Sb) 
which reflects the determination result is generated. 

Fig. 23 shows an example of the configuration of 
a table showing decoding flags F(Sb) for respective 
5 subbands Sb. Fig. 23 shows the table when the 

non-decoding subband number M = 3, and M subbands in 
turn from the uppermost one in the subband list of 
Fig. 23 are set to be non-decoding subbands. A process 
to be described later generates the table shown in 
10 Fig. 23 and stores the generated table data in the RAM 
51402. 

Fig. 24 is a flow chart showing the flow of the 
encoded moving image data decoding process by the image 
decoding apparatus according to this embodiment. Note 

15 that a program according to Fig. 24 is loaded onto the 
RAM 51402, and the image decoding apparatus according 
to this embodiment can execute the process according to 
the flow chart shown in Fig. 24 when the CPU 51401 
executes the loaded program. The image decoding 

20 process to be executed by the image decoding apparatus 
according to this embodiment will be described in more 
detail below with reference to the flow chart shown in 
Fig. 24. 

Initially, the bitplane decoder 5102 and inverse 
25 discrete wavelet transformer 5104 decode encoded image 
data of the sample frame, which is read out from the 
secondary storage device 206 by the code sequence read 
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unit 5101, and the decoding process time measurement 
unit 5106 measures decoding times Dt(SI) of subbands, 
which form the encoded Image data of the sample frame, 
and a decoding process time Dt as a time required until 
5 all subbands which form the encoded Image data of the 
sample freune are decoded and output (step S51900). 

The non-decoding subband determination unit 5107 
determines the non- decoding subband number M on the 
basis of the times measured in step S51900 (step 

10 S51901). If the actual decoding process time Dt ^ the 
target decoding time T, since the decoding time for one 
frame falls within the target decoding process time 
even if all subbands that form each frame are decoded, 
M = 0 (the non -decoding subband number = 0, i.e. , all 

15 subbands are to be decoded) is set. 

On the other hand, if Dt > T, the time required 
until image data for one frame is reconstructed and 
output cannot fall within the target decoding process 
time if all subbands that form one frame are decoded. 

20 Hence, a process for setting a non-decoding subband in 
turn from that of a lower resolution of the subbands 
which form one frame is executed. In this manner, 
since the number of subbands to be decoded can be 
decreased, the time required until image data for one 

25 frame is reconstructed and output can fall within the 
target decoding process time. 
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More specifically, the non- decoding subbands M is 
decreased in the order from subband number 0 to subband 
number 6 until Dt ^ T. More specifically, minimum M 
which satisfies: 



is obtained, and is set as the non- decoding subband 
number M (step S51901). 

With the above process, the non-decoding subband 
number M that may decode and output each frame within 

10 the target decoding process time is determined. Note 
that the above process determines the number of 
non-decoding subbands, i.e., the number of subbands 
which are not to be decoded. This process is 
equivalent to that for determining the number of 

15 subbands to be decoded. 

Next, the decoding processes of encoded image 
data of respective frames are executed in turn from the 
first frame (frame 1 in this case). Prior to these 
processes, a time different AT is reset to zero (step 

20 S51902). 

The non-decoding subband determination unit 5107 
determines decoding flags F(Sb) on the basis of the 
non-decoding subband number M determined in step S51901 
(step S51903). The process in step S51903 will be 

25 explained below taking the table of Fig. 23 as an 

example. For example, if M = 3 is determined in step 



5 
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S51901, flags F(Sb) corresponding to Sb = HH2, LH2, and 
HL2, i.e., three flags F(Sb) in turn from the uppermost 
one are set to be zero, and other flags F(Sb) are set 
to be 1. The table data generated in this way is 
5 stored in the RAM 510402, as described above. 

The code sequence read unit 5101 reads out 
encoded image data of respective frames from the 
secondary storage device 206, and outputs them to the 
next bit plane decoder 5102. The bit plane decoder 5102 

10 decodes bitplanes of only subbands to be decoded in the 
encoded image data of each frame input from the code 
sequence read unit 5101 by looking up the F(Sb) table 
data. Furthermore, the inverse discrete wavelet 
transformer 5104 executes the inverse discrete wavelet 

15 transformation process using the decoding result, and 
outputs the decoded result of each frame, i.e., image 
data of each frame, to the moving image data output 
unit 5105 (step S51904). 

The decoding process time measurement unit 5106 

20 measures a time Dt required for a series of processes 
in step S51904 (i.e. , a time required until image data 
for one frame is reconstructed and output in step 
S51904), and notifies the non-decoding subband 
determination unit 5107 of it (step S51905). 

25 The non-decoding subband determination unit 5107 

calculates the difference between the target decoding 
time T for one frame and the actually required decoding 
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process time Dt. and adds it to the held time 
difference AT (step S51906). This time difference AT 
accumulates the "difference between the target decoding 
time T for one frame and the actually required decoding 
5 process time Dt" every time encoded image data for one 
frame is decoded. 

Next, the non-decoding subband number M is 
updated according to the value AT (step S51907). That 
is, if AT is larger than a value Dt(M-l) x n, which is 

10 calculated and stored in the RAM 51402 in advance, 1 is 
subtracted from the value held by the variable M, i.e., 
the non- decoding subband number to decrease the value 
of the variable M. 

Note that n is a predetermined arbitrary value, 

15 and the non -decoding subband nximber becomes harder to 
change as n assumes a larger value. AT becomes larger 
than a predetermined threshold value (Dt(M-l) x n in 
this embodiment) when the sum total of the actually 
required decoding times Dt (Dt x G (G is the number of 

20 times of the decoding process executed so far) ) is 
smaller than the sum total of the target decoding 
process times T (T x G (G is the number of times of the 
decoding process executed so far)). In such case, it 
can be determined that there is a time margin for 

25 decoding some extra subbands. 

Hence, the value of the non -decoding subband 
number M is decreased by 1 (i.e. , the number of 
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subbands to be decoded is Increased by 1 ) to improve 
the decoded image quality. Note that the value to be 
subtracted from the non- decoding subband number is not 
limited to "1", and a value equal to or larger than "2" 
5 may be used depending on the threshold value used. 

Note that Dt(M-l) x n used as the threshold value 
in this embodiment is a rough standard of the required 
decoding time of a subband which may become a new 
subband to be decoded since the non-decoding subband 

10 number M is decreased by 1. 

Conversely, if AT is smaller than a value Lq (Lq 
< 0) which is calculated and stored in advance in the 
RAM 51402, 1 is added to M to increase the value. 

AT becomes smaller than a predetermined threshold 

15 value (Lq in this embodiment) when the sum total of the 
actually required decoding times Dt (Dt x G (G is the 
number of times of the decoding process executed so 
far)) is larger than the sum total of the target 
decoding process times T (T x G (G is the number of 

20 times of the decoding process executed so far)). In 
this case, it can be determined that the overall 
process delays. 

Therefore, the non-decoding subband number is 
increased to shorten the decoding time per frame. Note 

25 that the value M ranges from 0 to 7, and if M < 0 after 
the aforementioned update process, M=0;ifM>7, M= 
7. 
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The processes In steps S51904 to S51908 are 
repeated for all frames (or frames designated using the 
keyboard 51402, mouse 51405, and the like) of the 
moving image. If the above processes have been done 
5 for all the frames, this process ends; otherwise, the 
flow returns to step S51904 to repeat the subsequent 
processes . 

With the above process, the decoding process can 
be made by determining the number of subbands as 

10 decoding units. 

As described above, the image decoding apparatus 
according to this embodiment can estimate the required 
decoding time of each subband, and determines the 
non-decoding subband number M on the basis of this 

15 required decoding time and the difference between the 
time required for the decoding process for one frame 
and the target decoding time, thus controlling the 
decoding process time while suppressing visual problems 
of a playback image as much as possible. 

20 < Sixth Embodiment > 

In the fifth embodiment, the required decoding 
time for each subband is estimated and the non-decoding 
subband number is determined so as to control the 
decoding time for one frame to fall within a 

25 predetermined time. That is, a subband is used as a 

decoding unit. In this embodiment, in order to achieve 
the scune object as in the fifth embodiment, "bitplane" 
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is used as this decoding unit, a parameter (to be 
referred to as a Q factor hereinafter) used to control 
the decoded image quality step by step is set, and the 
required decoding time based on each Q factor is 
5 estimated to determine the number of non-decoding 
bitplanes. In the image decoding apparatus of this 
embodiment, Q factors of 10 levels, i.e., those from 0 
to 9, are specified, and the number of non-decoding 
bitplanes of each subband is determined for respective 

10 Q factors. An example of the Q factor will be 
explained later. 

The image decoding apparatus of this embodiment 
decodes encoded image data that has undergone subband 
decomposition using a real number type 5x3 filter. 

15 That is, in the aforementioned moving image encoding 

apparatus, a signal for selecting the real number type 
5x3 filter is input via the signal line 207 to encode 
moving image data. Also, identical quantization steps 
Delta(Sb) are used for all freunes upon encoding frame 

20 images . 

Furthermore, the basic arrangement of the image 
decoding apparatus according to this embodiment is the 
same as that of the fifth embodiment, i.e., the 
arrangement shown in Fig. 26. 
25 The image decoding apparatus according to this 

embodiment comprises the functional arrangement shown 
in Fig. 10. As shown in Fig. 10, the image decoding 
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apparatus according to this embodiment comprises a code 
sequence read unit 904, bltplane decoder 102, 
coefficient dequantlzer 901, Inverse discrete wavelet 
transformer 902, moving Image data output unit 105, 
5 decoding process time measurement unit 106, and 

non-decoding bltplane determination unit 903. Note 
that the arrangement shown In Fig. 10 may be 
Implemented by hardware. However, in this embodiment, 
the respective units shown In Fig. 10 are Implemented 

10 by a program which makes a computer Implement the 

functions of the respective units, and this program Is 
loaded from the external storage device 51407 or 
storage medium driver 51408, or from the external 
device via the I/F 51409 onto the RAM 51402. 

15 In Fig. 10, reference numeral 206 denotes a 

secondary storage device described above, from which 
encoded moving image data is input to this apparatus . 
Assume that this secondary storage device 206 
corresponds to the external storage device 51407 or 

20 storage medium driver 51408, or the external device 
connected to this apparatus via the I/F 51409, and 
encoded moving image data is loaded onto the RAM 51402 
from one of these devices. Note that this encoded 
moving image data is generated by the moving image 

25 encoding apparatus, as described above. 
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The process to be executed by the Image decoding 
apparatus according to this embodiment will be 
described below with reference to Fig. 10. 

The Image decoding apparatus according to this 
5 embodiment Initially extracts encoded Image data of an 
arbitrary freune from the encoded moving Image data to 
be decoded as a sample, decodes that data using various 
Q factor values , and measures decoding times according 
to the respective Q factor values . 

10 In general, encoded moving Image data Is decoded 

for each frame In that data. The code sequence read 
unit 904 shown In Fig. 10 reads out encoded data of a 
frame of Interest from the encoded moving Image data 
stored in the secondary storage device 206, and stores 

15 It In the RAM 51402. The encoded data for respective 
frames are basically read out In turn like frame 1, 
frame 2 , . . . , but this embodiment reads out encoded data 
of an arbitrary frame as a sample and stores It In the 
RAM 51402, so as to decode the encoded Image data of 

20 that frame as the sample using various Q factor values 
and to measure decoding times according to the 
respective Q factor values before the decoding 
processes of respective frames. In the following 
description, assume that the 60th frame Is used as this 

25 frame (to be referred to as a sample frame hereinafter) 
for the sake of simplicity, but the present invention 
is not limited to such specific frame. 
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Also, upon reading out encoded Image data of a 
frame, the code sequence read unit 904 executes a 
process for referring to the header of this encoded 
Image data, reading out a quantization step delta(Sb) 
5 of each subband from the header, and storing It In the 
RAM 51402. 

The bltplane decoder 102 reads out the encoded 
data stored In the RAM 51402 In the order of subbands 
and executes the same process as In the fifth 

10 embodiment to decode quantized transform coefficient 

data Q(Sb, x, y) . The process In the bltplane decoder 
102 Is opposite to that In the bltplane encoder 204 
shown In Fig . 1 . 

In the fifth embodiment, the non-decoding subband 

15 determination unit 5107 determines data F(Sb) 

Indicating If each subband Is to be decoded, and the 
bltplane decoder 5102 switches decodlng/non~decodlng 
for each subband according to F(Sb) upon executing the 
bltplane decoding process of each frame - 

20 In this embodiment, the non-decoding bltplane 

determination unit 903 determines a "non-decoding 
bltplane number ND(Sb)" Indicating the number of lower 
bits which are not to undergo bltplane decoding for 
each subband. ND(Sb) Is data Indicating that decoding 

25 processes of bltplanes for lower ND(Sb) bits of subband 
Sb are to be skipped. 
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Therefore, the bitplane decoder 102 can skip 
bitplane decoding processes for lower ND(Sb) bits of 
subband Sb with reference to this data ND(Sb) upon 
executing the bltplane decoding process of each frame. 
5 A process for obtaining such ND(Sb) will be described 
later. 

Note that ND(Sb) = 0 Is set for all subbands Sb 

of the sample frame which Is to be decoded first by the 

Image decoding apparatus of this embodiment, and 
10 bltplanes of all subbands are decoded. Assume that the 

ND(Sb) data for the sample frcime are loaded In advance 

onto the RAM 51402. 

The coefficient dequantlzer 901 reclaims 

coefficients C(Sb, x, y) of each subband based on the 
15 quantization step delta(Sb) determined for each subband 

and the quantized coefficient values Q(Sb, x, y) 

decoded by the bltplane decoder 102. 

The Inverse discrete wavelet transformer 902 

executes Inverse transformation of the wavelet 
20 transformation process In the discrete wavelet 

transfojnner 202 In Fig. 1 to reclaim data of the frame. 

In this embodiment , since encoded moving Image data 

generated using the real number type 5x3 filter for 

all the frames Is to be decoded. Inverse transformation 
25 corresponding to equations (3) and (4) above Is done. 

The moving Image data output unit 105 outputs 

reconstructed Image data output from the Inverse 
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discrete wavelet transformer 902 to an external device 
via the I/F 51409 or to the display device 51406. 
However, since the data of the sample frame which 
initially undergoes a plurality of decoding processes 
5 by the image decoding apparatus of this embodiment is 
to be decoded to estimate the decoding process time, 
each decoded frame data is discarded and is not output. 

The decoding process time measurement unit 106 
measures a decoding time for one frame required from 

10 the beginning of read of encoded image data for one 
frame by the code sequence read unit 904 from the 
secondary storage device 206 until the bitplane decoder 
102, coefficient dequantizer 901, and inverse discrete 
wavelet transformer 902 execute a decoding process of 

15 the encoded image data for one frame and image data is 
output to the moving image data output unit 105. In 
this case, the unit 106 changes the Q factor value from 
0 to 9, and measures decoding times for one frame in 
correspondence with respective cases. That is, the 

20 unit 106 measures a decoding time for one frame when 
the Q factor value is 0, that when the Q factor value 
is 1 , . . . , and that when the Q factor value is 9 . 

In this embodiment , since the sample frame is to 
be decoded first, the decoding process time measurement 

25 unit 106 measures a decoding time of the sample freime 
when the Q factor value is 0, that when the Q factor 
value is 1 , . . . , and that when the Q factor value is 9 . 
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Fig. 20 shows decoding times Dt'(Q) of the sample 
frame corresponding to respective Q factor values . 
Fields Dt'(O) to Dt'(9) store the actually measured 
times. The decoding process time measurement unit 106 
5 executes the process for measuring the respective 
decoding times to generate data of decoding times 
Dt'(Q) of the sample frame corresponding to the 
respective Q factor values, and stores them In the RAM 
51402. 

10 Upon completion of the process for obtaining data 

of decoding times Dt'(O) to Dt'{9) shown in Fig. 20, 
and storing them in the RAM 51402, the decoding process 
time measurement unit 106 then executes a process for 
calculating an Increment Dt(Q) of the decoding process 

15 time with respect to a change in Q factor value. 

More specifically, the unit 106 calculates Dt(Q) 

by: 

Dt(Q) = Dt'(Q+l) - Df(Q) (7) 
for 0 ^ Q ^ 8 

20 The decoding process time measurement unit 106 

stores calculated data Dt(0) to Dt(8) in the RAM 51402. 

Also, the decoding process time measurement unit 
106 measures a time required from the beginning of read 
of encoded image data for one frame by the code 

25 sequence read unit 904 from the secondary storage 

device 206 until the moving image data output unit 105 
outputs reconstructed frame data as a "time required to 
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decode encoded Image data for one frame". Since this 
time is equivalent to Dt'(O) above, the unit 106 
outputs this data to the non-decoding bitplane 
determination unit 903. 
5 The decoding process time measurement unit 106 

measures the time Dt (i.e., Dt"(Q)) for each of frames 
other than the first frame in the moving image, and 
outputs it to the non-decoding bitplane determination 
unit 903. 

10 The non-decoding bitplane determination unit 903 

executes a process for determining bitplanes which are 
not to be decoded of those of each subband which forms 
the encoded image data for one frame, so that the 
decoding process time Dt of the sample frame output 

15 from the decoding process time measurement unit 106 

becomes equal to or smaller than a value pre -stored in 
the RAM 51402 as an upper limit value of the time 
required to reclaim and output image data for one frame. 
Assume that the RAM 51402 holds, before this 

20 process, data of a variable Q used as an index, a 

variable T indicating the target decoding process time 
as the upper limit value of the decoding time for one 
frame, a variable AT indicating the time difference, 
and a table registered with the non- decoding bitplane 

25 numbers of respective subbands according to the Q 
factor values . 
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Fig. 8 shows an example of ttie configuration of 
the table registered with the non- decoding bitplane 
numbers of respective subbands according to the Q 
factor values. The table shown in Fig. 8 is prepared 
5 in advance and is stored in the RAM 51402. As can be 
seen with reference to the table shown in Fig. 8, for 
excunple, when "4" is used as the Q factor value, ND(Sb) 
=4 for subbands Sb = HH2 , HL2, and HL2 (LH2), i.e., 
decoding processes of bitplanes for lower 4 bits are to 
10 be skipped; ND(Sb) = 3 for subband Sb = HLl (LHl), i.e., 
decoding processes of bitplanes for lower 3 bits are to 
be skipped; and ND(Sb) = 1 for subband Sb = LL, i.e., a 
decoding process of a bitplane for lower 1 bit is to be 
skipped. 

15 Hence, in the process to be described below, the 

Q factor value is obtained, and the number of 
non-decoding bitplanes for each subband is obtained 
according to the obtained Q factor value. 

Fig. 25 is a flow chart showing the flow of the 

20 encoded moving image data decoding process by the image 
decoding apparatus according to this embodiment. Note 
that a program according to Fig. 25 is loaded onto the 
RAM 51402, and the image decoding apparatus according 
to this embodiment can execute the process according. to 

25 the flow chart shown in Fig. 25 when the CPU 51401 
executes the loaded progreun. The image decoding 
process to be executed by the image decoding apparatus 
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according to this embodiment will be described In more 
detail below with reference to the flow chart shown In 
Fig. 25. 

The code sequence read unit 904 reads out encoded 
5 Image data of the sample frame from the secondary 

storage device 206 (step S52001). The decoding process 
time measurement unit 106 resets the variable Q by 
substituting a value 9 in it (step S52002). The 
decoding process time measurement unit 106 repeats 
10 processes in steps S52003 to S52006 to be described 
below until the variable Q assumes a negative value. 

The bltplane decoder 102, coefficient dequantlzer 
901, and Inverse discrete wavelet transformer 90.2 
decode the encoded image data of the sample frame, 
15 which is read out by the code sequence read unit 904 in 
step S52001 (step S52003). This decoding process 
corresponds to that when the Q factor value (a value 
held by the variable Q) is 9. 

The decoding process time measurement unit 106 
20 measures a time required for the process in step S52003 
(step S52004). At this time, since the value of the 
variable Q is "9", the time measured in step S52004 
corresponds to Dt'(9) above. The decoding process time 
measurement unit 106 stores data of Dt"(9) measured in 
25 step S52004 in the RAM 51402 (step S52005) . 

The decoding process time measurement unit 106 
executes a process for substituting a difference 
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obtained by subtracting 1 from the value held by the 
variable Q in the variable Q again (step S52006). 
After the process in step S52006, the value held by the 
variable Q is "8". 

The decoding process time measurement unit 106 
checks if the value held by the variable Q is negative 
(step S52007). If the value held by the variable Q is 
not negative, the flow returns to step S52003 to repeat 
the above processes. In this case, since the value 
held by the variable Q is "8", and 8 > 0, the flow 
returns to step S52003. 

A decoding process for Q 8 is executed in step 
S52003, and a decoding time required for that process 
is measured in step S52004. At this time, since the 
value of the variable Q is "8", the time measured in 
step S52004 corresponds to Dt'(8) above. 

In step S52005, a process for calculating an 
increment Dt(Q) of the decoding process time with 
respect to a change in Q factor value is executed 
according to equation (7) above. This process for 
calculating the increment is done according to equation 
(7) by: 

Dt(8) = Dt" (9) - Dt' (8) (8) 
That is, Dt(8) is calculated using Df(8) 
measured in the current step S52004 and Dt'(9) measured 
in the previous step S52004. The same applies to all 
cases when Q < 9. In general, Dt(Q) is calculated 
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using Dt'(Q) measured in the current step S52004 and 
Dt'{Q+l) measured in the previous step S52004. 

More specifically, in step S52005 if Q = 9 , data 
of Dt"(9) is stored in the RAM 51402 without 
5 calculating the increment Dt(Q); if Q < 9, the 

increment Dt(Q) is calculated according to equation (7) 

The processes in steps S52003 to S52006 are 
repeated until Q < 0, When Q < 0, Dt(0) to Dt{8) are 
obtained, and the non- decoding bitplane determination 
10 unit 903 obtains the Q factor value used to decode 
respective frames of moving image data using these 
Dt(0) to Dt(8) (step S52208). 

If Dt'(O) measured in step S52004 when Q = 0 and 
the target decoding process time T meet Dt'(O) ^ T, 
15 since the decoding time for one frame falls within the 
target decoding process time even when all bitplanes of 
all subbands which form each frame are decoded, Q = o 
(the non-decoding bitplane number = 0, i.e., all 
bitplanes are to be decoded) is set. 
20 On the other hand, if Dt'(O) > T, the time 

required until image data for one frame is 
reconstructed and output cannot fall within the target 
decoding process time if all bitplanes of all subbands 
that form one frame are decoded. Hence, the Q factor 
25 value that allows the time required until image data 
for one frame to fall within the target decoding 
process time is obtained, and bitplanes other than 
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non-decoding bltplanes specified by the obtained Q 
factor value are decoded with reference to the table 
shown In Fig. 8, thus controlling the time required 
until Image data for one frame to fall within the 
5 target decoding process time. 

More specifically, minimum Q which satisfies: 

is calculated, and is set as the Q factor value to be 

used in the subsequent processes. 
10 With the above process, the Q factor value 

required to decode and output each frame within the 

target decoding process time can be obtained. 

Next , the decoding processes of encoded image 

data of respective frames are executed in turn from the 
15 first freune (freune 1 in this case). Prior to these 

processes, a time difference AT is reset to zero (step 

S52209) . 

The non-decoding bitplane determination unit 903 
determines the non- decoding bitplane number on the 

20 basis of the Q factor value obtained in step S52208 
(step S52210). As described above, this process is 
done by acquiring the number of non -decoding bltplanes 
of each subband according to the value Q obtained in 
step S52208 with reference to the data of the table 

25 shown in, e.g.. Fig. 8. 
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Note that the process in step S52210 determines 
the number of non-decoding bitplanes, i.e., the number 
of bitplanes which are not to be decoded. In other 
words, this process is equivalent to that for 
5 determining the number of bitplanes to be decoded. The 
determined number of non- decoding bitplanes for each 
subband is stored in the RAM 51402 as the data ND(Sb). 

The code sequence read unit 904 reads out encoded 
image data of respective frames from the secondary 

10 storage device 206, and outputs them to the next 
bitplane decoder 102. The bitplane decoder 5102 
decodes only bitplanes to be decoded of each subband in 
the encoded image data of each frame input from the 
code sequence read unit 904 with reference to the data 

15 ND(Sb). Furthermore, the inverse discrete wavelet 

transformer 902 executes the inverse discrete wavelet 
transformation process using the decoding result, and 
outputs the decoded result of each frame, i.e., image 
data of each frame, to the moving image data output 

20 unit 105 (step S52211). 

The decoding process time measurement unit 106 
measures a time Dt required for a series of processes 
in step S52211 (i.e., a time required until image data 
for one frame is reconstructed and output in step 

25 S52211), and notifies the non-decoding bitplane 
determination unit 903 of it (step S52212). 
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The non- decoding bit plane determination unit 903 
calculates the difference between the target decoding 
time T for one frame and the actually required decoding 
process time Dt, and adds It to the held time 
5 difference AT (step S52213). This time difference AT 
accumulates the "difference between the target decoding 
time T for one frame and the actually required decoding 
process time Dt" every time encoded image data for one 
frcime is decoded. 

10 The Q factor value (the value held by the 

variable Q) is updated according to the value AT (step 
S52214). That is, if AT is larger than a value Dt(Q-l) 
X n, which is calculated and stored in the RAM 51402 in 
advance, 1 is subtracted from the value held by the 

15 variable Q to decrease the value of the variable Q, 

Note that n is a predetermined arbitrary value, 
and the Q factor value becomes harder to change as n 
assumes a larger value. AT becomes larger than a 
predetermined threshold value (Dt(Q-l) x n in this 

20 embodiment) when the sum total of the actually required 
decoding times Dt (Dt x G (G is the number of times of 
the decoding process executed so far) ) is smaller than 
the sum total of the target decoding process times T (T 
X G (G is the number of times of the decoding process 

25 executed so far)). In such case, it can be determined 
that there is a time margin for decoding some extra 
bltplanes . 
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Hence, the value of the Q factor value Is 
decreased by 1 (i.e., the number of non-decoding 
bitplanes is increased by 1 on the basis of the table 
in Fig. 8) to improve the decoded image quality. Note 
5 that the value to be subtracted from the Q factor value 
is not limited to " 1 " , and a value equal to or larger 
than "2" may be used depending on the threshold value 
used. 

Conversely, if AT is smaller than a value Lq (Lq 

10 < 0) which is calculated and stored in advance in the 
RAM 51402, 1 is added to the value held by Q to 
increase the value. 

AT becomes smaller than a predetermined threshold 
value (Lq in this embodiment) when the sum total of the 

15 actually required decoding times Dt (Dt x G (G is the 
number of times of the decoding process executed so 
far)) is larger than the sum total of the target 
decoding process times T(TxG(Gis the number of 
times of the decoding process executed so far)). In 

20 this case, it can be determined that the overall 
process delays. 

Therefore, the Q factor value is increased to 
shorten the decoding time per frame, thus increasing 
the number of non-decoding bitplanes. Note that the 

25 value Q ranges from 0 to 9, and if Q < 0 after the 

aforementioned update process, Q=0;ifQ>9, Q=9. 
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The processes In steps S52210 to S52214 are 
repeated for all frcunes (or frames designated using the 
keyboard 51402, mouse 51405, and the like) of the 
moving image. If the above processes have been done 
5 for all the frames, this process ends; otherwise, the 
flow returns to step S52210 to repeat the subsequent 
processes . 

With the above process, the decoding process can 
be made by determining the number of bitplanes as 

10 decoding units . 

As described above, the image decoding apparatus 
according to this embodiment decodes the sample frsune 
by changing an image quality parameter to estimate 
required decoding times corresponding to the changed 

15 image quality parameter values, and determines the 

image quality parameter to be used on the basis of the 
required decoding times and the difference between the 
time required for the decoding process of one frame and 
the target decoding time, thus controlling the decoding 

20 process time while suppressing visual problems of a 
playback image as much as possible. 
<Modification> 

For excunple, in the fifth and sixth embodiments, 
bitplane encoding is done for respective subbands. 

25 Alternatively, each subband may be segmented into 
blocks , and bitplane encoding may be made for 
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respective blocks. Also, one bltplane may be encoded 
using a plurality of passes. 

The fifth embodiment uses subbands and the sixth 
embodiment uses bitplanes as decoding units . In 
5 addition, code blocks may be used, and the present 
invention is not limited to such specific decoding 
units in the fifth and sixth embodiments. 

In the example described in the fifth and sixth 
embodiments, MQ-Coder is used as the binary arithmetic 

10 encoding method. However, the present invention is not 
limited to such specific method. For example, 
arithmetic encoding methods other than MQ- Coder such as 
QM-Coder and the like may be adopted, and other binary 
encoding methods may be adopted as long as they are 

15 suited to encode a multi-context information source. 

The filters for subband decomposition are not 
limited to those in the fifth and sixth embodiments. 
For example other filters such as a real number type 9 
X 7 filter and the like may be used. Furthermore, the 

20 number of times of application of the filter is not 

limited to that in the fifth and sixth embodiments. In 
the fifth and sixth embodiments, linear discrete 
wavelet transformation is applied the same number of 
times in the horizontal and vertical directions, but 

25 need not always be applied the same number of times in 
the horizontal and vertical directions. 
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Moreover, the structure of the encoded moving 
Image data is not limited to the fifth and sixth 
embodiments. For example, the order of the code 
sequence, the storage format of additional information, 
5 and the like may be modified. For example, the present 
invention is suitable for use of JPEG2000 specified by 
ISO/IEC15444-1 as the frame data encoding method, and 
encoded data described in the JPEG2000 standard or that 
of Motion JPEG2000 specified by the JPEG2000 standard 

10 Part 3 may be used. 

In the fifth and sixth embodiments, one arbitrary 
frame is decoded to estimate the process time prior to 
the decoding process. Alternatively, the decoding 
process time may be estimated using a plurality of 

15 frames as samples, or the estimated decoding process 
time may be updated during actual decoding processes . 
<Other Embodiments > 

Note that the present invention may be applied to 
either a system constituted by a plurality of devices 

20 (e.g., a host computer, interface device, reader, 

printer, and the like), or an apparatus consisting of a 
single equipment (e.g., a copying machine, facsimile 
apparatus, or the like). Also, techniques described in 
the respective embodiments may be combined. 

25 The objects of the present invention are also 

achieved by supplying a recording medium (or storage 
medium) , which records a program code of a software 
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program that can implement the functions of the 
above-mentioned embodiments to the system or apparatus, 
and reading out and executing the program code stored 
in the recording medium by a computer (or a CPU or MPU) 
5 of the system or apparatus. In this case, the program 
code itself read out from the recording medium 
implements the functions of the above-mentioned 
embodiments, and the recording medium which stores the 
program code constitutes the present invention. The 

10 functions of the above-mentioned embodiments may be 

implemented not only by executing the readout program 
code by the computer but also by some or all of actual 
processing operations executed by an operating system 
(OS) running on the computer on the basis of an 

15 instruction of the program code. 

Furthermore, the functions of the above-mentioned 
embodiments may be implemented by some or all of actual 
processing operations executed by a CPU or the like 
arranged in a function extension card or a function 

20 extension unit, which is inserted in or connected to 

the computer, after the program code read out from the 
recording medium is written in a memory of the 
extension card or unit- When the present invention is 
applied to the recording medium, that recording mediiim 

25 stores the program codes corresponding to the 
aforementioned flow charts. 
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When the present invention is applied to the 
recording meditun, that recording medium stores program 
codes corresponding to the aforementioned flow charts. 

As many apparently widely different embodiments 
5 of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
claims . 
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